Add new SentenceTransformer model

README.md

@@ -12,68 +12,120 @@ tags:
 - loss:MultipleNegativesRankingLoss
 base_model: Snowflake/snowflake-arctic-embed-m-v1.5
 widget:
-- source_sentence: Where can I find older versions of the ZenML documentation?
+- source_sentence: How can I connect a GCP Image Builder to resources using ZenML?
   sentences:
-  - 'gister flavors.my_flavor.MyExperimentTrackerFlavorZenML resolves the flavor class
-    by taking the path where you initialized zenml (via zenml init) as the starting
-    point of resolution. Therefore, please ensure you follow the best practice of
-    initializing zenml at the root of your repository.
+  - "_run.steps[step_name]\n    whylogs_step.visualize()if __name__ == \"__main__\"\
+    :\n    visualize_statistics(\"data_loader\")\n    visualize_statistics(\"train_data_profiler\"\
+    , \"test_data_profiler\")\n\nPreviousEvidentlyNextDevelop a custom data validator\n\
+    \nLast updated 1 month ago"
+  - 'Implement a custom integration
 
 
-    If ZenML does not find an initialized ZenML repository in any parent directory,
-    it will default to the current working directory, but usually, it''s better to
-    not have to rely on this mechanism and initialize zenml at the root.
+    Creating an external integration and contributing to ZenML
 
 
-    Afterward, you should see the new flavor in the list of available flavors:
+    PreviousContribute to ZenMLNextOverview
 
 
-    zenml experiment-tracker flavor list
-
-
-    It is important to draw attention to when and how these base abstractions are
-    coming into play in a ZenML workflow.
-
-
-    The CustomExperimentTrackerFlavor class is imported and utilized upon the creation
-    of the custom flavor through the CLI.
-
-
-    The CustomExperimentTrackerConfig class is imported when someone tries to register/update
-    a stack component with this custom flavor. Especially, during the registration
-    process of the stack component, the config will be used to validate the values
-    given by the user. As Config objects are inherently pydantic objects, you can
-    also add your own custom validators here.
-
-
-    The CustomExperimentTracker only comes into play when the component is ultimately
-    in use.
-
-
-    The design behind this interaction lets us separate the configuration of the flavor
-    from its implementation. This way we can register flavors and components even
-    when the major dependencies behind their implementation are not installed in our
-    local setting (assuming the CustomExperimentTrackerFlavor and the CustomExperimentTrackerConfig
-    are implemented in a different module/path than the actual CustomExperimentTracker).
-
-
-    PreviousWeights & BiasesNextModel Deployers
-
-
-    Last updated 21 days ago'
-  - 'ZenML - Bridging the gap between ML & Ops
-
-
-    Legacy Docs
-
-
-    Bleeding EdgeLegacy Docs0.67.0
-
-
-    🧙‍♂️Find older version our docs
-
-
-    Powered by GitBook'
+    Last updated 4 months ago'
+  - "--connector <CONNECTOR_ID>\n\nExample Command Output$ zenml image-builder connect\
+    \ gcp-image-builder --connector gcp-generic\nSuccessfully connected image builder\
+    \ `gcp-image-builder` to the following resources:\n┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┓\n\
+    ┃             CONNECTOR ID             │ CONNECTOR NAME │ CONNECTOR TYPE │ RESOURCE\
+    \ TYPE  │ RESOURCE NAMES ┃\n┠──────────────────────────────────────┼────────────────┼────────────────┼────────────────┼────────────────┨\n\
+    ┃ bfdb657d-d808-47e7-9974-9ba6e4919d83 │ gcp-generic    │ \U0001F535 gcp     \
+    \    │ \U0001F535 gcp-generic │ zenml-core     ┃\n┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┛\n\
+    \nAs a final step, you can use the GCP Image Builder in a ZenML Stack:\n\n# Register\
+    \ and set a stack with the new image builder\nzenml stack register <STACK_NAME>\
+    \ -i <IMAGE_BUILDER_NAME> ... --set\n\nWhen you register the GCP Image Builder,\
+    \ you can generate a GCP Service Account Key, save it to a local file and then\
+    \ reference it in the Image Builder configuration.\n\nThis method has the advantage\
+    \ that you don't need to install and configure the GCP CLI on your host, but it's\
+    \ still not as secure as using a GCP Service Connector and the stack component\
+    \ configuration is not portable to other hosts.\n\nFor this method, you need to\
+    \ create a user-managed GCP service account, and grant it privileges to access\
+    \ the Cloud Build API and to run Cloud Builder jobs (e.g. the Cloud Build Editor\
+    \ IAM role.\n\nWith the service account key downloaded to a local file, you can\
+    \ register the GCP Image Builder as follows:\n\nzenml image-builder register <IMAGE_BUILDER_NAME>\
+    \ \\\n    --flavor=gcp \\\n    --project=<GCP_PROJECT_ID> \\\n    --service_account_path=<PATH_TO_SERVICE_ACCOUNT_KEY>\
+    \ \\\n    --cloud_builder_image=<BUILDER_IMAGE_NAME> \\\n    --network=<DOCKER_NETWORK>\
+    \ \\\n    --build_timeout=<BUILD_TIMEOUT_IN_SECONDS>"
+- source_sentence: How do I register and activate a ZenML stack with a new GCP Image
+    Builder while ensuring proper authentication?
+  sentences:
+  - "oad the returned whylogs profile to WhyLabs, e.g.:import pandas as pd\nfrom whylogs.core\
+    \ import DatasetProfileView\nimport whylogs as why\nfrom zenml import step\nfrom\
+    \ zenml.integrations.whylogs.flavors.whylogs_data_validator_flavor import (\n\
+    \    WhylogsDataValidatorSettings,\n)\n\nwhylogs_settings = WhylogsDataValidatorSettings(\n\
+    \    enable_whylabs=True, dataset_id=\"<WHYLABS_DATASET_ID>\"\n)\n\n@step(\n \
+    \   settings={\n        \"data_validator\": whylogs_settings\n    }\n)\ndef data_profiler(\n\
+    \        dataset: pd.DataFrame,\n) -> DatasetProfileView:\n    \"\"\"Custom data\
+    \ profiler step with whylogs\n\nArgs:\n        dataset: a Pandas DataFrame\n\n\
+    Returns:\n        Whylogs Profile generated for the dataset\n    \"\"\"\n\n# validation\
+    \ pre-processing (e.g. dataset preparation) can take place here\n\nresults = why.log(dataset)\n\
+    \    profile = results.profile()\n\n# validation post-processing (e.g. interpret\
+    \ results, take actions) can happen here\n\nreturn profile.view()\n\nVisualizing\
+    \ whylogs Profiles\n\nYou can view visualizations of the whylogs profiles generated\
+    \ by your pipeline steps directly in the ZenML dashboard by clicking on the respective\
+    \ artifact in the pipeline run DAG.\n\nAlternatively, if you are running inside\
+    \ a Jupyter notebook, you can load and render the whylogs profiles using the artifact.visualize()\
+    \ method, e.g.:\n\nfrom zenml.client import Client\n\ndef visualize_statistics(\n\
+    \    step_name: str, reference_step_name: Optional[str] = None\n) -> None:\n \
+    \   \"\"\"Helper function to visualize whylogs statistics from step artifacts.\n\
+    \nArgs:\n        step_name: step that generated and returned a whylogs profile\n\
+    \        reference_step_name: an optional second step that generated a whylogs\n\
+    \            profile to use for data drift visualization where two whylogs\n \
+    \           profiles are required.\n    \"\"\"\n    pipe = Client().get_pipeline(pipeline=\"\
+    data_profiling_pipeline\")\n    whylogs_step = pipe.last_run.steps[step_name]\n\
+    \    whylogs_step.visualize()"
+  - "ogsDataValidatorSettings,\n)\nfrom zenml import step@step(\n    settings={\n\
+    \        \"data_validator\": WhylogsDataValidatorSettings(\n            enable_whylabs=True,\
+    \ dataset_id=\"model-1\"\n        )\n    }\n)\ndef data_loader() -> Tuple[\n \
+    \   Annotated[pd.DataFrame, \"data\"],\n    Annotated[DatasetProfileView, \"profile\"\
+    ]\n]:\n    \"\"\"Load the diabetes dataset.\"\"\"\n    X, y = datasets.load_diabetes(return_X_y=True,\
+    \ as_frame=True)\n\n# merge X and y together\n    df = pd.merge(X, y, left_index=True,\
+    \ right_index=True)\n\nprofile = why.log(pandas=df).profile().view()\n    return\
+    \ df, profile\n\nHow do you use it?\n\nWhylogs's profiling functions take in a\
+    \ pandas.DataFrame dataset generate a DatasetProfileView object containing all\
+    \ the relevant information extracted from the dataset.\n\nThere are three ways\
+    \ you can use whylogs in your ZenML pipelines that allow different levels of flexibility:\n\
+    \ninstantiate, configure and insert the standard WhylogsProfilerStep shipped with\
+    \ ZenML into your pipelines. This is the easiest way and the recommended approach,\
+    \ but can only be customized through the supported step configuration parameters.\n\
+    \ncall the data validation methods provided by the whylogs Data Validator in your\
+    \ custom step implementation. This method allows for more flexibility concerning\
+    \ what can happen in the pipeline step, but you are still limited to the functionality\
+    \ implemented in the Data Validator.\n\nuse the whylogs library directly in your\
+    \ custom step implementation. This gives you complete freedom in how you are using\
+    \ whylogs's features.\n\nYou can visualize whylogs profiles in Jupyter notebooks\
+    \ or view them directly in the ZenML dashboard.\n\nThe whylogs standard step"
+  - " build to finish. More information: Build Timeout.We can register the image builder\
+    \ and use it in our active stack:\n\nzenml image-builder register <IMAGE_BUILDER_NAME>\
+    \ \\\n    --flavor=gcp \\\n    --cloud_builder_image=<BUILDER_IMAGE_NAME> \\\n\
+    \    --network=<DOCKER_NETWORK> \\\n    --build_timeout=<BUILD_TIMEOUT_IN_SECONDS>\n\
+    \n# Register and activate a stack with the new image builder\nzenml stack register\
+    \ <STACK_NAME> -i <IMAGE_BUILDER_NAME> ... --set\n\nYou also need to set up authentication\
+    \ required to access the Cloud Build GCP services.\n\nAuthentication Methods\n\
+    \nIntegrating and using a GCP Image Builder in your pipelines is not possible\
+    \ without employing some form of authentication. If you're looking for a quick\
+    \ way to get started locally, you can use the Local Authentication method. However,\
+    \ the recommended way to authenticate to the GCP cloud platform is through a GCP\
+    \ Service Connector. This is particularly useful if you are configuring ZenML\
+    \ stacks that combine the GCP Image Builder with other remote stack components\
+    \ also running in GCP.\n\nThis method uses the implicit GCP authentication available\
+    \ in the environment where the ZenML code is running. On your local machine, this\
+    \ is the quickest way to configure a GCP Image Builder. You don't need to supply\
+    \ credentials explicitly when you register the GCP Image Builder, as it leverages\
+    \ the local credentials and configuration that the Google Cloud CLI stores on\
+    \ your local machine. However, you will need to install and set up the Google\
+    \ Cloud CLI on your machine as a prerequisite, as covered in the Google Cloud\
+    \ documentation , before you register the GCP Image Builder.\n\nStacks using the\
+    \ GCP Image Builder set up with local authentication are not portable across environments.\
+    \ To make ZenML pipelines fully portable, it is recommended to use a GCP Service\
+    \ Connector to authenticate your GCP Image Builder to the GCP cloud platform."
+- source_sentence: How can I register and set a stack with a new image builder using
+    ZenML?
+  sentences:
   - 'ZenML - Bridging the gap between ML & Ops
 
 
@@ -87,104 +139,104 @@ widget:
 
 
     Powered by GitBook'
-- source_sentence: Where can I find older versions of the ZenML documentation?
+  - "> \\\n    --build_timeout=<BUILD_TIMEOUT_IN_SECONDS># Register and set a stack\
+    \ with the new image builder\nzenml stack register <STACK_NAME> -i <IMAGE_BUILDER_NAME>\
+    \ ... --set\n\nCaveats\n\nAs described in this Google Cloud Build documentation\
+    \ page, Google Cloud Build uses containers to execute the build steps which are\
+    \ automatically attached to a network called cloudbuild that provides some Application\
+    \ Default Credentials (ADC), that allow the container to be authenticated and\
+    \ therefore use other GCP services.\n\nBy default, the GCP Image Builder is executing\
+    \ the build command of the ZenML Pipeline Docker image with the option --network=cloudbuild,\
+    \ so the ADC provided by the cloudbuild network can also be used in the build.\
+    \ This is useful if you want to install a private dependency from a GCP Artifact\
+    \ Registry, but you will also need to use a custom base parent image with the\
+    \ keyrings.google-artifactregistry-auth installed, so pip can connect and authenticate\
+    \ in the private artifact registry to download the dependency.\n\nFROM zenmldocker/zenml:latest\n\
+    \nRUN pip install keyrings.google-artifactregistry-auth\n\nThe above Dockerfile\
+    \ uses zenmldocker/zenml:latest as a base image, but is recommended to change\
+    \ the tag to specify the ZenML version and Python version like 0.33.0-py3.10.\n\
+    \nPreviousKaniko Image BuilderNextDevelop a Custom Image Builder\n\nLast updated\
+    \ 21 days ago"
+  - "res Spark to handle the resource configuration.\"\"\"def _backend_configuration(\n\
+    \            self,\n            spark_config: SparkConf,\n            step_config:\
+    \ \"StepConfiguration\",\n    ) -> None:\n        \"\"\"Configures Spark to handle\
+    \ backends like YARN, Mesos or Kubernetes.\"\"\"\n\ndef _io_configuration(\n \
+    \           self,\n            spark_config: SparkConf\n    ) -> None:\n     \
+    \   \"\"\"Configures Spark to handle different input/output sources.\"\"\"\n\n\
+    def _additional_configuration(\n            self,\n            spark_config: SparkConf\n\
+    \    ) -> None:\n        \"\"\"Appends the user-defined configuration parameters.\"\
+    \"\"\n\ndef _launch_spark_job(\n            self,\n            spark_config: SparkConf,\n\
+    \            entrypoint_command: List[str]\n    ) -> None:\n        \"\"\"Generates\
+    \ and executes a spark-submit command.\"\"\"\n\ndef launch(\n            self,\n\
+    \            info: \"StepRunInfo\",\n            entrypoint_command: List[str],\n\
+    \    ) -> None:\n        \"\"\"Launches the step on Spark.\"\"\"\n\nUnder the\
+    \ base configuration, you will see the main configuration parameters:\n\nmaster\
+    \ is the master URL for the cluster where Spark will run. You might see different\
+    \ schemes for this URL with varying cluster managers such as Mesos, YARN, or Kubernetes.\n\
+    \ndeploy_mode can either be 'cluster' (default) or 'client' and it decides where\
+    \ the driver node of the application will run.\n\nsubmit_args is the JSON string\
+    \ of a dictionary, which will be used to define additional parameters if required\
+    \ ( Spark has a wide variety of parameters, thus including them all in a single\
+    \ class was deemed unnecessary.).\n\nIn addition to this configuration, the launch\
+    \ method of the step operator gets additional configuration parameters from the\
+    \ DockerSettings and ResourceSettings. As a result, the overall configuration\
+    \ happens in 4 base methods:\n\n_resource_configuration translates the ZenML ResourceSettings\
+    \ object to Spark's own resource configuration.\n\n_backend_configuration is responsible\
+    \ for cluster-manager-specific configuration."
+- source_sentence: How can I install ZenML with support for a local dashboard, and
+    what precautions should I take when installing on a Mac with Apple Silicon?
   sentences:
-  - 'Whylogs
-
-
-    How to collect and visualize statistics to track changes in your pipelines'' data
-    with whylogs/WhyLabs profiling.
-
-
-    The whylogs/WhyLabs Data Validator flavor provided with the ZenML integration
-    uses whylogs and WhyLabs to generate and track data profiles, highly accurate
-    descriptive representations of your data. The profiles can be used to implement
-    automated corrective actions in your pipelines, or to render interactive representations
-    for further visual interpretation, evaluation and documentation.
-
-
-    When would you want to use it?
-
+  - ' visit our PyPi package page.
 
-    Whylogs is an open-source library that analyzes your data and creates statistical
-    summaries called whylogs profiles. Whylogs profiles can be processed in your pipelines
-    and visualized locally or uploaded to the WhyLabs platform, where more in depth
-    analysis can be carried out. Even though whylogs also supports other data types,
-    the ZenML whylogs integration currently only works with tabular data in pandas.DataFrame
-    format.
 
+    Running with Dockerzenml is also available as a Docker image hosted publicly on
+    DockerHub. Use the following command to get started in a bash environment with
+    zenml available:
 
-    You should use the whylogs/WhyLabs Data Validator when you need the following
-    data validation features that are possible with whylogs and WhyLabs:
 
+    docker run -it zenmldocker/zenml /bin/bash
 
-    Data Quality: validate data quality in model inputs or in a data pipeline
 
+    If you would like to run the ZenML server with Docker:
 
-    Data Drift: detect data drift in model input features
-
-
-    Model Drift: Detect training-serving skew, concept drift, and model performance
-    degradation
-
-
-    You should consider one of the other Data Validator flavors if you need a different
-    set of data validation features.
-
-
-    How do you deploy it?
-
-
-    The whylogs Data Validator flavor is included in the whylogs ZenML integration,
-    you need to install it on your local machine to be able to register a whylogs
-    Data Validator and add it to your stack:
-
-
-    zenml integration install whylogs -y
-
-
-    If you don''t need to connect to the WhyLabs platform to upload and store the
-    generated whylogs data profiles, the Data Validator stack component does not require
-    any configuration parameters. Adding it to a stack is as simple as running e.g.:'
-  - 'ZenML - Bridging the gap between ML & Ops
 
+    docker run -it -d -p 8080:8080 zenmldocker/zenml-server
 
-    Legacy Docs
 
+    Deploying the server
 
-    Bleeding EdgeLegacy Docs0.67.0
 
+    Though ZenML can run entirely as a pip package on a local system, complete with
+    the dashboard. You can do this easily:
 
-    🧙‍♂️Find older version our docs
 
+    pip install "zenml[server]"
 
-    Powered by GitBook'
-  - 'ZenML - Bridging the gap between ML & Ops
+    zenml up  # opens the dashboard locally
 
 
-    Legacy Docs
+    However, advanced ZenML features are dependent on a centrally-deployed ZenML server
+    accessible to other MLOps stack components. You can read more about it here.
 
 
-    Bleeding EdgeLegacy Docs0.67.0
+    For the deployment of ZenML, you have the option to either self-host it or register
+    for a free ZenML Pro account.
 
 
-    🧙‍♂️Find older version our docs
+    PreviousIntroductionNextCore concepts
 
 
-    Powered by GitBook'
-- source_sentence: How can I install ZenML with support for a local dashboard, and
-    what precautions should I take when installing on a Mac with Apple Silicon?
-  sentences:
-  - 'Finetuning LLMs with ZenML
+    Last updated 20 days ago'
+  - 'Evaluation and metrics
 
 
-    Finetune LLMs for specific tasks or to improve performance and cost.
+    Track how your RAG pipeline improves using evaluation and metrics.
 
 
-    PreviousEvaluating finetuned embeddingsNextSet up a project repository
+    PreviousBasic RAG inference pipelineNextEvaluation in 65 lines of code
 
 
-    Last updated 6 months ago'
+    Last updated 4 months ago'
   - '🧙Installation
 
 
@@ -270,188 +322,70 @@ widget:
 
 
     Running with Docker'
-  - "se you decide to switch to another Data Validator.All you have to do is call\
-    \ the whylogs Data Validator methods when you need to interact with whylogs to\
-    \ generate data profiles. You may optionally enable whylabs logging to automatically\
-    \ upload the returned whylogs profile to WhyLabs, e.g.:\n\nimport pandas as pd\n\
-    from whylogs.core import DatasetProfileView\nfrom zenml.integrations.whylogs.data_validators.whylogs_data_validator\
-    \ import (\n    WhylogsDataValidator,\n)\nfrom zenml.integrations.whylogs.flavors.whylogs_data_validator_flavor\
-    \ import (\n    WhylogsDataValidatorSettings,\n)\nfrom zenml import step\n\nwhylogs_settings\
-    \ = WhylogsDataValidatorSettings(\n    enable_whylabs=True, dataset_id=\"<WHYLABS_DATASET_ID>\"\
-    \n)\n\n@step(\n    settings={\n        \"data_validator\": whylogs_settings\n\
-    \    }\n)\ndef data_profiler(\n        dataset: pd.DataFrame,\n) -> DatasetProfileView:\n\
-    \    \"\"\"Custom data profiler step with whylogs\n\nArgs:\n        dataset: a\
-    \ Pandas DataFrame\n\nReturns:\n        Whylogs profile generated for the data\n\
-    \    \"\"\"\n\n# validation pre-processing (e.g. dataset preparation) can take\
-    \ place here\n\ndata_validator = WhylogsDataValidator.get_active_data_validator()\n\
-    \    profile = data_validator.data_profiling(\n        dataset,\n    )\n    #\
-    \ optionally upload the profile to WhyLabs, if WhyLabs credentials are configured\n\
-    \    data_validator.upload_profile_view(profile)\n\n# validation post-processing\
-    \ (e.g. interpret results, take actions) can happen here\n\nreturn profile\n\n\
-    Have a look at the complete list of methods and parameters available in the WhylogsDataValidator\
-    \ API in the SDK docs.\n\nCall whylogs directly\n\nYou can use the whylogs library\
-    \ directly in your custom pipeline steps, and only leverage ZenML's capability\
-    \ of serializing, versioning and storing the DatasetProfileView objects in its\
-    \ Artifact Store. You may optionally enable whylabs logging to automatically upload\
-    \ the returned whylogs profile to WhyLabs, e.g.:"
-- source_sentence: How can I finetune embeddings using Sentence Transformers as described
-    in the ZenML documentation?
+- source_sentence: How does the KubernetesSparkStepOperator utilize the PipelineDockerImageBuilder
+    class to manage Docker images for Spark jobs on Kubernetes?
   sentences:
-  - 'Evaluation and metrics
+  - 'ZenML - Bridging the gap between ML & Ops
 
 
-    Track how your RAG pipeline improves using evaluation and metrics.
+    Legacy Docs
 
 
-    PreviousBasic RAG inference pipelineNextEvaluation in 65 lines of code
+    Bleeding EdgeLegacy Docs0.67.0
 
 
-    Last updated 4 months ago'
-  - ":\n        \"\"\"Abstract method to deploy a model.\"\"\"@staticmethod\n    @abstractmethod\n\
-    \    def get_model_server_info(\n            service: BaseService,\n    ) -> Dict[str,\
-    \ Optional[str]]:\n        \"\"\"Give implementation-specific way to extract relevant\
-    \ model server\n        properties for the user.\"\"\"\n\n@abstractmethod\n  \
-    \  def perform_stop_model(\n        self,\n        service: BaseService,\n   \
-    \     timeout: int = DEFAULT_DEPLOYMENT_START_STOP_TIMEOUT,\n        force: bool\
-    \ = False,\n    ) -> BaseService:\n        \"\"\"Abstract method to stop a model\
-    \ server.\"\"\"\n\n@abstractmethod\n    def perform_start_model(\n        self,\n\
-    \        service: BaseService,\n        timeout: int = DEFAULT_DEPLOYMENT_START_STOP_TIMEOUT,\n\
-    \    ) -> BaseService:\n        \"\"\"Abstract method to start a model server.\"\
-    \"\"\n\n@abstractmethod\n    def perform_delete_model(\n        self,\n      \
-    \  service: BaseService,\n        timeout: int = DEFAULT_DEPLOYMENT_START_STOP_TIMEOUT,\n\
-    \        force: bool = False,\n    ) -> None:\n        \"\"\"Abstract method to\
-    \ delete a model server.\"\"\"\n\nclass BaseModelDeployerFlavor(Flavor):\n   \
-    \ \"\"\"Base class for model deployer flavors.\"\"\"\n\n@property\n    @abstractmethod\n\
-    \    def name(self):\n        \"\"\"Returns the name of the flavor.\"\"\"\n\n\
-    @property\n    def type(self) -> StackComponentType:\n        \"\"\"Returns the\
-    \ flavor type.\n\nReturns:\n            The flavor type.\n        \"\"\"\n   \
-    \     return StackComponentType.MODEL_DEPLOYER\n\n@property\n    def config_class(self)\
-    \ -> Type[BaseModelDeployerConfig]:\n        \"\"\"Returns `BaseModelDeployerConfig`\
-    \ config class.\n\nReturns:\n                The config class.\n        \"\"\"\
-    \n        return BaseModelDeployerConfig\n\n@property\n    @abstractmethod\n \
-    \   def implementation_class(self) -> Type[BaseModelDeployer]:\n        \"\"\"\
-    The class that implements the model deployer.\"\"\"\n\nThis is a slimmed-down\
-    \ version of the base implementation which aims to highlight the abstraction layer.\
-    \ In order to see the full implementation and get the complete docstrings, please\
-    \ check the SDK docs .\n\nBuilding your own model deployers"
-  - 'Finetuning embeddings with Sentence Transformers
-
-
-    Finetune embeddings with Sentence Transformers.
-
-
-    PreviousSynthetic data generationNextEvaluating finetuned embeddings
-
-
-    Last updated 1 month ago'
-- source_sentence: How does ZenML utilize type annotations in step outputs to enhance
-    data handling between pipeline steps?
-  sentences:
-  - "ator which runs Steps with Spark on Kubernetes.\"\"\"def _backend_configuration(\n\
-    \            self,\n            spark_config: SparkConf,\n            step_config:\
-    \ \"StepConfiguration\",\n    ) -> None:\n        \"\"\"Configures Spark to run\
-    \ on Kubernetes.\"\"\"\n        # Build and push the image\n        docker_image_builder\
-    \ = PipelineDockerImageBuilder()\n        image_name = docker_image_builder.build_and_push_docker_image(...)\n\
-    \n# Adjust the spark configuration\n        spark_config.set(\"spark.kubernetes.container.image\"\
-    , image_name)\n        ...\n\nFor Kubernetes, there are also some additional important\
-    \ configuration parameters:\n\nnamespace is the namespace under which the driver\
-    \ and executor pods will run.\n\nservice_account is the service account that will\
-    \ be used by various Spark components (to create and watch the pods).\n\nAdditionally,\
-    \ the _backend_configuration method is adjusted to handle the Kubernetes-specific\
-    \ configuration.\n\nWhen to use it\n\nYou should use the Spark step operator:\n\
-    \nwhen you are dealing with large amounts of data.\n\nwhen you are designing a\
-    \ step that can benefit from distributed computing paradigms in terms of time\
-    \ and resources.\n\nHow to deploy it\n\nTo use the KubernetesSparkStepOperator\
-    \ you will need to setup a few things first:\n\nRemote ZenML server: See the deployment\
-    \ guide for more information.\n\nKubernetes cluster: There are many ways to deploy\
-    \ a Kubernetes cluster using different cloud providers or on your custom infrastructure.\
-    \ For AWS, you can follow the Spark EKS Setup Guide below.\n\nSpark EKS Setup\
-    \ Guide\n\nThe following guide will walk you through how to spin up and configure\
-    \ a Amazon Elastic Kubernetes Service with Spark on it:\n\nEKS Kubernetes Cluster\n\
-    \nFollow this guide to create an Amazon EKS cluster role.\n\nFollow this guide\
-    \ to create an Amazon EC2 node role.\n\nGo to the IAM website, and select Roles\
-    \ to edit both roles.\n\nAttach the AmazonRDSFullAccess and AmazonS3FullAccess\
-    \ policies to both roles.\n\nGo to the EKS website.\n\nMake sure the correct region\
-    \ is selected on the top right."
-  - "\U0001F5C4️Handle Data/Artifacts\n\nStep outputs in ZenML are stored in the artifact\
-    \ store. This enables caching, lineage and auditability. Using type annotations\
-    \ helps with transparency, passing data between steps, and serializing/des\n\n\
-    For best results, use type annotations for your outputs. This is good coding practice\
-    \ for transparency, helps ZenML handle passing data between steps, and also enables\
-    \ ZenML to serialize and deserialize (referred to as 'materialize' in ZenML) the\
-    \ data.\n\n@step\ndef load_data(parameter: int) -> Dict[str, Any]:\n\n# do something\
-    \ with the parameter here\n\ntraining_data = [[1, 2], [3, 4], [5, 6]]\n    labels\
-    \ = [0, 1, 0]\n    return {'features': training_data, 'labels': labels}\n\n@step\n\
-    def train_model(data: Dict[str, Any]) -> None:\n    total_features = sum(map(sum,\
-    \ data['features']))\n    total_labels = sum(data['labels'])\n    \n    # Train\
-    \ some model here\n    \n    print(f\"Trained model using {len(data['features'])}\
-    \ data points. \"\n          f\"Feature sum is {total_features}, label sum is\
-    \ {total_labels}\")\n\n@pipeline  \ndef simple_ml_pipeline(parameter: int):\n\
-    \    dataset = load_data(parameter=parameter)  # Get the output \n    train_model(dataset)\
-    \  # Pipe the previous step output into the downstream step\n\nIn this code, we\
-    \ define two steps: load_data and train_model. The load_data step takes an integer\
-    \ parameter and returns a dictionary containing training data and labels. The\
-    \ train_model step receives the dictionary from load_data, extracts the features\
-    \ and labels, and trains a model (not shown here).\n\nFinally, we define a pipeline\
-    \ simple_ml_pipeline that chains the load_data and train_model steps together.\
-    \ The output from load_data is passed as input to train_model, demonstrating how\
-    \ data flows between steps in a ZenML pipeline.\n\nPreviousDisable colorful loggingNextHow\
-    \ ZenML stores data\n\nLast updated 4 months ago"
-  - ' your GCP Image Builder to the GCP cloud platform.To set up the GCP Image Builder
-    to authenticate to GCP and access the GCP Cloud Build services, it is recommended
-    to leverage the many features provided by the GCP Service Connector such as auto-configuration,
-    best security practices regarding long-lived credentials and reusing the same
-    credentials across multiple stack components.
-
-
-    If you don''t already have a GCP Service Connector configured in your ZenML deployment,
-    you can register one using the interactive CLI command. You also have the option
-    to configure a GCP Service Connector that can be used to access more than just
-    the GCP Cloud Build service:
-
-
-    zenml service-connector register --type gcp -i
-
-
-    A non-interactive CLI example that leverages the Google Cloud CLI configuration
-    on your local machine to auto-configure a GCP Service Connector for the GCP Cloud
-    Build service:
-
-
-    zenml service-connector register <CONNECTOR_NAME> --type gcp --resource-type gcp-generic
-    --resource-name <GCS_BUCKET_NAME> --auto-configure
-
-
-    Example Command Output
-
-
-    $ zenml service-connector register gcp-generic --type gcp --resource-type gcp-generic
-    --auto-configure
-
-    Successfully registered service connector `gcp-generic` with access to the following
-    resources:
-
-    ┏━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┓
-
-    ┃ RESOURCE TYPE  │ RESOURCE NAMES ┃
-
-    ┠────────────────┼────────────────┨
-
-    ┃ 🔵 gcp-generic │ zenml-core     ┃
-
-    ┗━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┛
-
-
-    Note: Please remember to grant the entity associated with your GCP credentials
-    permissions to access the Cloud Build API and to run Cloud Builder jobs (e.g.
-    the Cloud Build Editor IAM role). The GCP Service Connector supports many different
-    authentication methods with different levels of security and convenience. You
-    should pick the one that best fits your use case.
-
-
-    If you already have one or more GCP Service Connectors configured in your ZenML
-    deployment, you can check which of them can be used to access generic GCP resources
-    like the GCP Image Builder required for your GCP Image Builder by running e.g.:'
+    🧙‍♂️Find older version our docs
+
+
+    Powered by GitBook'
+  - "nsible for cluster-manager-specific configuration._io_configuration is a critical\
+    \ method. Even though we have materializers, Spark might require additional packages\
+    \ and configuration to work with a specific filesystem. This method is used as\
+    \ an interface to provide this configuration.\n\n_additional_configuration takes\
+    \ the submit_args, converts, and appends them to the overall configuration.\n\n\
+    Once the configuration is completed, _launch_spark_job comes into play. This takes\
+    \ the completed configuration and runs a Spark job on the given master URL with\
+    \ the specified deploy_mode. By default, this is achieved by creating and executing\
+    \ a spark-submit command.\n\nWarning\n\nIn its first iteration, the pre-configuration\
+    \ with _io_configuration method is only effective when it is paired with an S3ArtifactStore\
+    \ (which has an authentication secret). When used with other artifact store flavors,\
+    \ you might be required to provide additional configuration through the submit_args.\n\
+    \nStack Component: KubernetesSparkStepOperator\n\nThe KubernetesSparkStepOperator\
+    \ is implemented by subclassing the base SparkStepOperator and uses the PipelineDockerImageBuilder\
+    \ class to build and push the required Docker images.\n\nfrom typing import Optional\n\
+    \nfrom zenml.integrations.spark.step_operators.spark_step_operator import (\n\
+    \    SparkStepOperatorConfig\n)\n\nclass KubernetesSparkStepOperatorConfig(SparkStepOperatorConfig):\n\
+    \    \"\"\"Config for the Kubernetes Spark step operator.\"\"\"\n\nnamespace:\
+    \ Optional[str] = None\n    service_account: Optional[str] = None\n\nfrom pyspark.conf\
+    \ import SparkConf\n\nfrom zenml.utils.pipeline_docker_image_builder import PipelineDockerImageBuilder\n\
+    from zenml.integrations.spark.step_operators.spark_step_operator import (\n  \
+    \  SparkStepOperator\n)\n\nclass KubernetesSparkStepOperator(SparkStepOperator):\n\
+    \    \"\"\"Step operator which runs Steps with Spark on Kubernetes.\"\"\""
+  - "ngs/python/Dockerfile -u 0 build\n\nConfiguring RBACAdditionally, you may need\
+    \ to create the several resources in Kubernetes in order to give Spark access\
+    \ to edit/manage your driver executor pods.\n\nTo do so, create a file called\
+    \ rbac.yaml with the following content:\n\napiVersion: v1\nkind: Namespace\nmetadata:\n\
+    \  name: spark-namespace\n---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n\
+    \  name: spark-service-account\n  namespace: spark-namespace\n---\napiVersion:\
+    \ rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\nmetadata:\n  name: spark-role\n\
+    \  namespace: spark-namespace\nsubjects:\n  - kind: ServiceAccount\n    name:\
+    \ spark-service-account\n    namespace: spark-namespace\nroleRef:\n  kind: ClusterRole\n\
+    \  name: edit\n  apiGroup: rbac.authorization.k8s.io\n---\n\nAnd then execute\
+    \ the following command to create the resources:\n\naws eks --region=$REGION update-kubeconfig\
+    \ --name=$EKS_CLUSTER_NAME\n\nkubectl create -f rbac.yaml\n\nLastly, note down\
+    \ the namespace and the name of the service account since you will need them when\
+    \ registering the stack component in the next step.\n\nHow to use it\n\nTo use\
+    \ the KubernetesSparkStepOperator, you need:\n\nthe ZenML spark integration. If\
+    \ you haven't installed it already, run\n\nzenml integration install spark\n\n\
+    Docker installed and running.\n\nA remote artifact store as part of your stack.\n\
+    \nA remote container registry as part of your stack.\n\nA Kubernetes cluster deployed.\n\
+    \nWe can then register the step operator and use it in our active stack:\n\nzenml\
+    \ step-operator register spark_step_operator \\\n\t--flavor=spark-kubernetes \\\
+    \n\t--master=k8s://$EKS_API_SERVER_ENDPOINT \\\n\t--namespace=<SPARK_KUBERNETES_NAMESPACE>\
+    \ \\\n\t--service_account=<SPARK_KUBERNETES_SERVICE_ACCOUNT>\n\n# Register the\
+    \ stack\nzenml stack register spark_stack \\\n    -o default \\\n    -s spark_step_operator\
+    \ \\\n    -a spark_artifact_store \\\n    -c spark_container_registry \\\n   \
+    \ -i local_builder \\\n    --set"
 pipeline_tag: sentence-similarity
 library_name: sentence-transformers
 metrics:
@@ -536,7 +470,7 @@ model-index:
       value: 0.75
       name: Cosine Accuracy@1
     - type: cosine_accuracy@3
-      value: 1.0
+      value: 0.75
       name: Cosine Accuracy@3
     - type: cosine_accuracy@5
       value: 1.0
@@ -548,7 +482,7 @@ model-index:
       value: 0.75
       name: Cosine Precision@1
     - type: cosine_precision@3
-      value: 0.3333333333333333
+      value: 0.25
       name: Cosine Precision@3
     - type: cosine_precision@5
       value: 0.2
@@ -560,7 +494,7 @@ model-index:
       value: 0.75
       name: Cosine Recall@1
     - type: cosine_recall@3
-      value: 1.0
+      value: 0.75
       name: Cosine Recall@3
     - type: cosine_recall@5
       value: 1.0
@@ -569,13 +503,13 @@ model-index:
       value: 1.0
       name: Cosine Recall@10
     - type: cosine_ndcg@10
-      value: 0.875
+      value: 0.8576691395183482
       name: Cosine Ndcg@10
     - type: cosine_mrr@10
-      value: 0.8333333333333334
+      value: 0.8125
       name: Cosine Mrr@10
     - type: cosine_map@100
-      value: 0.8333333333333334
+      value: 0.8125
       name: Cosine Map@100
   - task:
       type: information-retrieval
@@ -691,9 +625,9 @@ This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [S
 
 ### Model Description
 - **Model Type:** Sentence Transformer
-- **Base model:** [Snowflake/snowflake-arctic-embed-m-v1.5](https://huggingface.co/Snowflake/snowflake-arctic-embed-m-v1.5) <!-- at revision 3b5a16eaf17e47bd997da998988dce5877a57092 -->
+- **Base model:** [Snowflake/snowflake-arctic-embed-m-v1.5](https://huggingface.co/Snowflake/snowflake-arctic-embed-m-v1.5) <!-- at revision 4d7418a980f09b897b7e08dcd981603eefde0e3f -->
 - **Maximum Sequence Length:** 512 tokens
-- **Output Dimensionality:** 768 tokens
+- **Output Dimensionality:** 768 dimensions
 - **Similarity Function:** Cosine Similarity
 - **Training Dataset:**
     - json
@@ -734,9 +668,9 @@ from sentence_transformers import SentenceTransformer
 model = SentenceTransformer("zenml/finetuned-snowflake-arctic-embed-m-v1.5")
 # Run inference
 sentences = [
-    'How does ZenML utilize type annotations in step outputs to enhance data handling between pipeline steps?',
-    '🗄️Handle Data/Artifacts\n\nStep outputs in ZenML are stored in the artifact store. This enables caching, lineage and auditability. Using type annotations helps with transparency, passing data between steps, and serializing/des\n\nFor best results, use type annotations for your outputs. This is good coding practice for transparency, helps ZenML handle passing data between steps, and also enables ZenML to serialize and deserialize (referred to as \'materialize\' in ZenML) the data.\n\n@step\ndef load_data(parameter: int) -> Dict[str, Any]:\n\n# do something with the parameter here\n\ntraining_data = [[1, 2], [3, 4], [5, 6]]\n    labels = [0, 1, 0]\n    return {\'features\': training_data, \'labels\': labels}\n\n@step\ndef train_model(data: Dict[str, Any]) -> None:\n    total_features = sum(map(sum, data[\'features\']))\n    total_labels = sum(data[\'labels\'])\n    \n    # Train some model here\n    \n    print(f"Trained model using {len(data[\'features\'])} data points. "\n          f"Feature sum is {total_features}, label sum is {total_labels}")\n\n@pipeline  \ndef simple_ml_pipeline(parameter: int):\n    dataset = load_data(parameter=parameter)  # Get the output \n    train_model(dataset)  # Pipe the previous step output into the downstream step\n\nIn this code, we define two steps: load_data and train_model. The load_data step takes an integer parameter and returns a dictionary containing training data and labels. The train_model step receives the dictionary from load_data, extracts the features and labels, and trains a model (not shown here).\n\nFinally, we define a pipeline simple_ml_pipeline that chains the load_data and train_model steps together. The output from load_data is passed as input to train_model, demonstrating how data flows between steps in a ZenML pipeline.\n\nPreviousDisable colorful loggingNextHow ZenML stores data\n\nLast updated 4 months ago',
-    " your GCP Image Builder to the GCP cloud platform.To set up the GCP Image Builder to authenticate to GCP and access the GCP Cloud Build services, it is recommended to leverage the many features provided by the GCP Service Connector such as auto-configuration, best security practices regarding long-lived credentials and reusing the same credentials across multiple stack components.\n\nIf you don't already have a GCP Service Connector configured in your ZenML deployment, you can register one using the interactive CLI command. You also have the option to configure a GCP Service Connector that can be used to access more than just the GCP Cloud Build service:\n\nzenml service-connector register --type gcp -i\n\nA non-interactive CLI example that leverages the Google Cloud CLI configuration on your local machine to auto-configure a GCP Service Connector for the GCP Cloud Build service:\n\nzenml service-connector register <CONNECTOR_NAME> --type gcp --resource-type gcp-generic --resource-name <GCS_BUCKET_NAME> --auto-configure\n\nExample Command Output\n\n$ zenml service-connector register gcp-generic --type gcp --resource-type gcp-generic --auto-configure\nSuccessfully registered service connector `gcp-generic` with access to the following resources:\n┏━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┓\n┃ RESOURCE TYPE  │ RESOURCE NAMES ┃\n┠────────────────┼────────────────┨\n┃ 🔵 gcp-generic │ zenml-core     ┃\n┗━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┛\n\nNote: Please remember to grant the entity associated with your GCP credentials permissions to access the Cloud Build API and to run Cloud Builder jobs (e.g. the Cloud Build Editor IAM role). The GCP Service Connector supports many different authentication methods with different levels of security and convenience. You should pick the one that best fits your use case.\n\nIf you already have one or more GCP Service Connectors configured in your ZenML deployment, you can check which of them can be used to access generic GCP resources like the GCP Image Builder required for your GCP Image Builder by running e.g.:",
+    'How does the KubernetesSparkStepOperator utilize the PipelineDockerImageBuilder class to manage Docker images for Spark jobs on Kubernetes?',
+    'nsible for cluster-manager-specific configuration._io_configuration is a critical method. Even though we have materializers, Spark might require additional packages and configuration to work with a specific filesystem. This method is used as an interface to provide this configuration.\n\n_additional_configuration takes the submit_args, converts, and appends them to the overall configuration.\n\nOnce the configuration is completed, _launch_spark_job comes into play. This takes the completed configuration and runs a Spark job on the given master URL with the specified deploy_mode. By default, this is achieved by creating and executing a spark-submit command.\n\nWarning\n\nIn its first iteration, the pre-configuration with _io_configuration method is only effective when it is paired with an S3ArtifactStore (which has an authentication secret). When used with other artifact store flavors, you might be required to provide additional configuration through the submit_args.\n\nStack Component: KubernetesSparkStepOperator\n\nThe KubernetesSparkStepOperator is implemented by subclassing the base SparkStepOperator and uses the PipelineDockerImageBuilder class to build and push the required Docker images.\n\nfrom typing import Optional\n\nfrom zenml.integrations.spark.step_operators.spark_step_operator import (\n    SparkStepOperatorConfig\n)\n\nclass KubernetesSparkStepOperatorConfig(SparkStepOperatorConfig):\n    """Config for the Kubernetes Spark step operator."""\n\nnamespace: Optional[str] = None\n    service_account: Optional[str] = None\n\nfrom pyspark.conf import SparkConf\n\nfrom zenml.utils.pipeline_docker_image_builder import PipelineDockerImageBuilder\nfrom zenml.integrations.spark.step_operators.spark_step_operator import (\n    SparkStepOperator\n)\n\nclass KubernetesSparkStepOperator(SparkStepOperator):\n    """Step operator which runs Steps with Spark on Kubernetes."""',
+    "ngs/python/Dockerfile -u 0 build\n\nConfiguring RBACAdditionally, you may need to create the several resources in Kubernetes in order to give Spark access to edit/manage your driver executor pods.\n\nTo do so, create a file called rbac.yaml with the following content:\n\napiVersion: v1\nkind: Namespace\nmetadata:\n  name: spark-namespace\n---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  name: spark-service-account\n  namespace: spark-namespace\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\nmetadata:\n  name: spark-role\n  namespace: spark-namespace\nsubjects:\n  - kind: ServiceAccount\n    name: spark-service-account\n    namespace: spark-namespace\nroleRef:\n  kind: ClusterRole\n  name: edit\n  apiGroup: rbac.authorization.k8s.io\n---\n\nAnd then execute the following command to create the resources:\n\naws eks --region=$REGION update-kubeconfig --name=$EKS_CLUSTER_NAME\n\nkubectl create -f rbac.yaml\n\nLastly, note down the namespace and the name of the service account since you will need them when registering the stack component in the next step.\n\nHow to use it\n\nTo use the KubernetesSparkStepOperator, you need:\n\nthe ZenML spark integration. If you haven't installed it already, run\n\nzenml integration install spark\n\nDocker installed and running.\n\nA remote artifact store as part of your stack.\n\nA remote container registry as part of your stack.\n\nA Kubernetes cluster deployed.\n\nWe can then register the step operator and use it in our active stack:\n\nzenml step-operator register spark_step_operator \\\n\t--flavor=spark-kubernetes \\\n\t--master=k8s://$EKS_API_SERVER_ENDPOINT \\\n\t--namespace=<SPARK_KUBERNETES_NAMESPACE> \\\n\t--service_account=<SPARK_KUBERNETES_SERVICE_ACCOUNT>\n\n# Register the stack\nzenml stack register spark_stack \\\n    -o default \\\n    -s spark_step_operator \\\n    -a spark_artifact_store \\\n    -c spark_container_registry \\\n    -i local_builder \\\n    --set",
 ]
 embeddings = model.encode(sentences)
 print(embeddings.shape)
@@ -777,92 +711,27 @@ You can finetune this model on your own dataset.
 ### Metrics
 
 #### Information Retrieval
-* Dataset: `dim_384`
-* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
-
-| Metric              | Value      |
-|:--------------------|:-----------|
-| cosine_accuracy@1   | 0.75       |
-| cosine_accuracy@3   | 1.0        |
-| cosine_accuracy@5   | 1.0        |
-| cosine_accuracy@10  | 1.0        |
-| cosine_precision@1  | 0.75       |
-| cosine_precision@3  | 0.3333     |
-| cosine_precision@5  | 0.2        |
-| cosine_precision@10 | 0.1        |
-| cosine_recall@1     | 0.75       |
-| cosine_recall@3     | 1.0        |
-| cosine_recall@5     | 1.0        |
-| cosine_recall@10    | 1.0        |
-| cosine_ndcg@10      | 0.875      |
-| cosine_mrr@10       | 0.8333     |
-| **cosine_map@100**  | **0.8333** |
-
-#### Information Retrieval
-* Dataset: `dim_256`
-* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
 
-| Metric              | Value      |
-|:--------------------|:-----------|
-| cosine_accuracy@1   | 0.75       |
-| cosine_accuracy@3   | 1.0        |
-| cosine_accuracy@5   | 1.0        |
-| cosine_accuracy@10  | 1.0        |
-| cosine_precision@1  | 0.75       |
-| cosine_precision@3  | 0.3333     |
-| cosine_precision@5  | 0.2        |
-| cosine_precision@10 | 0.1        |
-| cosine_recall@1     | 0.75       |
-| cosine_recall@3     | 1.0        |
-| cosine_recall@5     | 1.0        |
-| cosine_recall@10    | 1.0        |
-| cosine_ndcg@10      | 0.875      |
-| cosine_mrr@10       | 0.8333     |
-| **cosine_map@100**  | **0.8333** |
-
-#### Information Retrieval
-* Dataset: `dim_128`
-* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
-
-| Metric              | Value      |
-|:--------------------|:-----------|
-| cosine_accuracy@1   | 0.75       |
-| cosine_accuracy@3   | 0.75       |
-| cosine_accuracy@5   | 1.0        |
-| cosine_accuracy@10  | 1.0        |
-| cosine_precision@1  | 0.75       |
-| cosine_precision@3  | 0.25       |
-| cosine_precision@5  | 0.2        |
-| cosine_precision@10 | 0.1        |
-| cosine_recall@1     | 0.75       |
-| cosine_recall@3     | 0.75       |
-| cosine_recall@5     | 1.0        |
-| cosine_recall@10    | 1.0        |
-| cosine_ndcg@10      | 0.8577     |
-| cosine_mrr@10       | 0.8125     |
-| **cosine_map@100**  | **0.8125** |
-
-#### Information Retrieval
-* Dataset: `dim_64`
+* Datasets: `dim_384`, `dim_256`, `dim_128` and `dim_64`
 * Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
 
-| Metric              | Value      |
-|:--------------------|:-----------|
-| cosine_accuracy@1   | 0.75       |
-| cosine_accuracy@3   | 1.0        |
-| cosine_accuracy@5   | 1.0        |
-| cosine_accuracy@10  | 1.0        |
-| cosine_precision@1  | 0.75       |
-| cosine_precision@3  | 0.3333     |
-| cosine_precision@5  | 0.2        |
-| cosine_precision@10 | 0.1        |
-| cosine_recall@1     | 0.75       |
-| cosine_recall@3     | 1.0        |
-| cosine_recall@5     | 1.0        |
-| cosine_recall@10    | 1.0        |
-| cosine_ndcg@10      | 0.875      |
-| cosine_mrr@10       | 0.8333     |
-| **cosine_map@100**  | **0.8333** |
+| Metric              | dim_384   | dim_256    | dim_128    | dim_64    |
+|:--------------------|:----------|:-----------|:-----------|:----------|
+| cosine_accuracy@1   | 0.75      | 0.75       | 0.75       | 0.75      |
+| cosine_accuracy@3   | 1.0       | 0.75       | 0.75       | 1.0       |
+| cosine_accuracy@5   | 1.0       | 1.0        | 1.0        | 1.0       |
+| cosine_accuracy@10  | 1.0       | 1.0        | 1.0        | 1.0       |
+| cosine_precision@1  | 0.75      | 0.75       | 0.75       | 0.75      |
+| cosine_precision@3  | 0.3333    | 0.25       | 0.25       | 0.3333    |
+| cosine_precision@5  | 0.2       | 0.2        | 0.2        | 0.2       |
+| cosine_precision@10 | 0.1       | 0.1        | 0.1        | 0.1       |
+| cosine_recall@1     | 0.75      | 0.75       | 0.75       | 0.75      |
+| cosine_recall@3     | 1.0       | 0.75       | 0.75       | 1.0       |
+| cosine_recall@5     | 1.0       | 1.0        | 1.0        | 1.0       |
+| cosine_recall@10    | 1.0       | 1.0        | 1.0        | 1.0       |
+| **cosine_ndcg@10**  | **0.875** | **0.8577** | **0.8577** | **0.875** |
+| cosine_mrr@10       | 0.8333    | 0.8125     | 0.8125     | 0.8333    |
+| cosine_map@100      | 0.8333    | 0.8125     | 0.8125     | 0.8333    |
 
 <!--
 ## Bias, Risks and Limitations
@@ -889,13 +758,13 @@ You can finetune this model on your own dataset.
   |         | positive                                                                           | anchor                                                                               |
   |:--------|:-----------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|
   | type    | string                                                                             | string                                                                               |
-  | details | <ul><li>min: 13 tokens</li><li>mean: 23.14 tokens</li><li>max: 38 tokens</li></ul> | <ul><li>min: 31 tokens</li><li>mean: 311.83 tokens</li><li>max: 512 tokens</li></ul> |
+  | details | <ul><li>min: 13 tokens</li><li>mean: 23.11 tokens</li><li>max: 38 tokens</li></ul> | <ul><li>min: 31 tokens</li><li>mean: 299.64 tokens</li><li>max: 512 tokens</li></ul> |
 * Samples:
-  | positive                                                                                                                                                   | anchor                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
-  |:-----------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-  | <code>What are the necessary steps to deploy the KubernetesSparkStepOperator, and what configurations are required for running Spark on Kubernetes?</code> | <code>ator which runs Steps with Spark on Kubernetes."""def _backend_configuration(<br>            self,<br>            spark_config: SparkConf,<br>            step_config: "StepConfiguration",<br>    ) -> None:<br>        """Configures Spark to run on Kubernetes."""<br>        # Build and push the image<br>        docker_image_builder = PipelineDockerImageBuilder()<br>        image_name = docker_image_builder.build_and_push_docker_image(...)<br><br># Adjust the spark configuration<br>        spark_config.set("spark.kubernetes.container.image", image_name)<br>        ...<br><br>For Kubernetes, there are also some additional important configuration parameters:<br><br>namespace is the namespace under which the driver and executor pods will run.<br><br>service_account is the service account that will be used by various Spark components (to create and watch the pods).<br><br>Additionally, the _backend_configuration method is adjusted to handle the Kubernetes-specific configuration.<br><br>When to use it<br><br>You should use the Spark step operator:<br><br>when you are dealing with large amounts of data.<br><br>when you are designing a step that can benefit from distributed computing paradigms in terms of time and resources.<br><br>How to deploy it<br><br>To use the KubernetesSparkStepOperator you will need to setup a few things first:<br><br>Remote ZenML server: See the deployment guide for more information.<br><br>Kubernetes cluster: There are many ways to deploy a Kubernetes cluster using different cloud providers or on your custom infrastructure. For AWS, you can follow the Spark EKS Setup Guide below.<br><br>Spark EKS Setup Guide<br><br>The following guide will walk you through how to spin up and configure a Amazon Elastic Kubernetes Service with Spark on it:<br><br>EKS Kubernetes Cluster<br><br>Follow this guide to create an Amazon EKS cluster role.<br><br>Follow this guide to create an Amazon EC2 node role.<br><br>Go to the IAM website, and select Roles to edit both roles.<br><br>Attach the AmazonRDSFullAccess and AmazonS3FullAccess policies to both roles.<br><br>Go to the EKS website.<br><br>Make sure the correct region is selected on the top right.</code> |
-  | <code>How do I set up a GCP Service Connector within ZenML to authenticate and access GCP Cloud Build services?</code>                                     | <code> your GCP Image Builder to the GCP cloud platform.To set up the GCP Image Builder to authenticate to GCP and access the GCP Cloud Build services, it is recommended to leverage the many features provided by the GCP Service Connector such as auto-configuration, best security practices regarding long-lived credentials and reusing the same credentials across multiple stack components.<br><br>If you don't already have a GCP Service Connector configured in your ZenML deployment, you can register one using the interactive CLI command. You also have the option to configure a GCP Service Connector that can be used to access more than just the GCP Cloud Build service:<br><br>zenml service-connector register --type gcp -i<br><br>A non-interactive CLI example that leverages the Google Cloud CLI configuration on your local machine to auto-configure a GCP Service Connector for the GCP Cloud Build service:<br><br>zenml service-connector register <CONNECTOR_NAME> --type gcp --resource-type gcp-generic --resource-name <GCS_BUCKET_NAME> --auto-configure<br><br>Example Command Output<br><br>$ zenml service-connector register gcp-generic --type gcp --resource-type gcp-generic --auto-configure<br>Successfully registered service connector `gcp-generic` with access to the following resources:<br>┏━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┓<br>┃ RESOURCE TYPE  │ RESOURCE NAMES ┃<br>┠────────────────┼────────────────┨<br>┃ 🔵 gcp-generic │ zenml-core     ┃<br>┗━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┛<br><br>Note: Please remember to grant the entity associated with your GCP credentials permissions to access the Cloud Build API and to run Cloud Builder jobs (e.g. the Cloud Build Editor IAM role). The GCP Service Connector supports many different authentication methods with different levels of security and convenience. You should pick the one that best fits your use case.<br><br>If you already have one or more GCP Service Connectors configured in your ZenML deployment, you can check which of them can be used to access generic GCP resources like the GCP Image Builder required for your GCP Image Builder by running e.g.:</code>                                                                           |
-  | <code>How do I register and activate a ZenML stack with a new GCP Image Builder while ensuring proper authentication?</code>                               | <code> build to finish. More information: Build Timeout.We can register the image builder and use it in our active stack:<br><br>zenml image-builder register <IMAGE_BUILDER_NAME> \<br>    --flavor=gcp \<br>    --cloud_builder_image=<BUILDER_IMAGE_NAME> \<br>    --network=<DOCKER_NETWORK> \<br>    --build_timeout=<BUILD_TIMEOUT_IN_SECONDS><br><br># Register and activate a stack with the new image builder<br>zenml stack register <STACK_NAME> -i <IMAGE_BUILDER_NAME> ... --set<br><br>You also need to set up authentication required to access the Cloud Build GCP services.<br><br>Authentication Methods<br><br>Integrating and using a GCP Image Builder in your pipelines is not possible without employing some form of authentication. If you're looking for a quick way to get started locally, you can use the Local Authentication method. However, the recommended way to authenticate to the GCP cloud platform is through a GCP Service Connector. This is particularly useful if you are configuring ZenML stacks that combine the GCP Image Builder with other remote stack components also running in GCP.<br><br>This method uses the implicit GCP authentication available in the environment where the ZenML code is running. On your local machine, this is the quickest way to configure a GCP Image Builder. You don't need to supply credentials explicitly when you register the GCP Image Builder, as it leverages the local credentials and configuration that the Google Cloud CLI stores on your local machine. However, you will need to install and set up the Google Cloud CLI on your machine as a prerequisite, as covered in the Google Cloud documentation , before you register the GCP Image Builder.<br><br>Stacks using the GCP Image Builder set up with local authentication are not portable across environments. To make ZenML pipelines fully portable, it is recommended to use a GCP Service Connector to authenticate your GCP Image Builder to the GCP cloud platform.</code>                                                                                                                                                                                                                                           |
+  | positive                                                                                                                                                                        | anchor                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+  |:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+  | <code>How does the ZenML BaseService registry manage serialization and re-creation of configurations for BaseService instances as part of the remote model server setup?</code> | <code>e details of the deployment process from the user.It needs to act as a ZenML BaseService registry, where every BaseService instance is used as an internal representation of a remote model server (see the find_model_server abstract method). To achieve this, it must be able to re-create the configuration of a BaseService from information that is persisted externally, alongside, or even as part of the remote model server configuration itself. For example, for model servers that are implemented as Kubernetes resources, the BaseService instances can be serialized and saved as Kubernetes resource annotations. This allows the model deployer to keep track of all externally running model servers and to re-create their corresponding BaseService instance representations at any given time. The model deployer also defines methods that implement basic life-cycle management on remote model servers outside the coverage of a pipeline (see stop_model_server , start_model_server and delete_model_server)....</code>                                                                                     |
+  | <code>How do you ensure the MyExperimentTrackerFlavor is properly registered and available in ZenML?</code>                                                                     | <code>gister flavors.my_flavor.MyExperimentTrackerFlavorZenML resolves the flavor class by taking the path where you initialized zenml (via zenml init) as the starting point of resolution. Therefore, please ensure you follow the best practice of initializing zenml at the root of your repository.<br><br>If ZenML does not find an initialized ZenML repository in any parent directory, it will default to the current working directory, but usually, it's better to not have to rely on this mechanism and initialize zenml at the root.<br><br>Afterward, you should see the new flavor in the list of available flavors:<br><br>zenml experiment-tracker flavor list<br><br>It is important to draw attention to when and how these base abstractions are coming into play in a ZenML workflow.<br><br>The CustomExperimentTrackerFlavor class is imported and utilized upon the creation of the custom flavor through the CLI.<br><br>The CustomExperimentTrackerConfig class is imported when someone tries to register/update a stack component with this custom fl...</code>                                                 |
+  | <code>How do you load and profile a dataset using the Whylogs data validator in ZenML?</code>                                                                                   | <code>ogsDataValidatorSettings,<br>)<br>from zenml import step@step(<br>    settings={<br>        "data_validator": WhylogsDataValidatorSettings(<br>            enable_whylabs=True, dataset_id="model-1"<br>        )<br>    }<br>)<br>def data_loader() -> Tuple[<br>    Annotated[pd.DataFrame, "data"],<br>    Annotated[DatasetProfileView, "profile"]<br>]:<br>    """Load the diabetes dataset."""<br>    X, y = datasets.load_diabetes(return_X_y=True, as_frame=True)<br><br># merge X and y together<br>    df = pd.merge(X, y, left_index=True, right_index=True)<br><br>profile = why.log(pandas=df).profile().view()<br>    return df, profile<br><br>How do you use it?<br><br>Whylogs's profiling functions take in a pandas.DataFrame dataset generate a DatasetProfileView object containing all the relevant information extracted from the dataset.<br><br>There are three ways you can use whylogs in your ZenML pipelines that allow different levels of flexibility:<br><br>instantiate, configure and insert the standard WhylogsProfilerStep shipped with ZenML into your pipelines. This is the easiest ...</code> |
 * Loss: [<code>MatryoshkaLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters:
   ```json
   {
@@ -989,7 +858,7 @@ You can finetune this model on your own dataset.
 - `dataloader_num_workers`: 0
 - `dataloader_prefetch_factor`: None
 - `past_index`: -1
-- `disable_tqdm`: True
+- `disable_tqdm`: False
 - `remove_unused_columns`: True
 - `label_names`: None
 - `load_best_model_at_end`: True
@@ -1043,30 +912,30 @@ You can finetune this model on your own dataset.
 - `optim_target_modules`: None
 - `batch_eval_metrics`: False
 - `eval_on_start`: False
-- `use_liger_kernel`: False
 - `eval_use_gather_object`: False
+- `prompts`: None
 - `batch_sampler`: no_duplicates
 - `multi_dataset_batch_sampler`: proportional
 
 </details>
 
 ### Training Logs
-| Epoch   | Step  | dim_384_cosine_map@100 | dim_256_cosine_map@100 | dim_128_cosine_map@100 | dim_64_cosine_map@100 |
+| Epoch   | Step  | dim_384_cosine_ndcg@10 | dim_256_cosine_ndcg@10 | dim_128_cosine_ndcg@10 | dim_64_cosine_ndcg@10 |
 |:-------:|:-----:|:----------------------:|:----------------------:|:----------------------:|:---------------------:|
-| **1.0** | **1** | **0.8333**             | **0.8333**             | **0.8125**             | **0.8333**            |
-| 2.0     | 3     | 0.8333                 | 0.8333                 | 0.8125                 | 0.8333                |
-| 3.0     | 4     | 0.8333                 | 0.8333                 | 0.8125                 | 0.8333                |
+| **1.0** | **1** | **0.875**              | **0.875**              | **0.8577**             | **0.875**             |
+| 2.0     | 3     | 0.875                  | 0.8577                 | 0.8577                 | 0.875                 |
+| 3.0     | 4     | 0.875                  | 0.8577                 | 0.8577                 | 0.875                 |
 
 * The bold row denotes the saved checkpoint.
 
 ### Framework Versions
-- Python: 3.11.9
-- Sentence Transformers: 3.2.0
-- Transformers: 4.45.2
-- PyTorch: 2.5.0+cu124
-- Accelerate: 1.0.1
-- Datasets: 3.0.1
-- Tokenizers: 0.20.1
+- Python: 3.11.10
+- Sentence Transformers: 3.3.1
+- Transformers: 4.43.1
+- PyTorch: 2.5.1+cu124
+- Accelerate: 1.1.1
+- Datasets: 3.1.0
+- Tokenizers: 0.19.1
 
 ## Citation
 

config.json

@@ -19,7 +19,7 @@
   "pad_token_id": 0,
   "position_embedding_type": "absolute",
   "torch_dtype": "float32",
-  "transformers_version": "4.45.2",
+  "transformers_version": "4.43.1",
   "type_vocab_size": 2,
   "use_cache": true,
   "vocab_size": 30522

config_sentence_transformers.json

@@ -1,12 +1,12 @@
 {
   "__version__": {
-    "sentence_transformers": "3.2.0",
-    "transformers": "4.45.2",
-    "pytorch": "2.5.0+cu124"
+    "sentence_transformers": "3.3.1",
+    "transformers": "4.43.1",
+    "pytorch": "2.5.1+cu124"
   },
   "prompts": {
     "query": "Represent this sentence for searching relevant passages: "
   },
   "default_prompt_name": null,
-  "similarity_fn_name": null
+  "similarity_fn_name": "cosine"
 }