zenml
/

finetuned-snowflake-arctic-embed-m-v1.5

@@ -12,25 +12,142 @@ tags:
 - loss:MultipleNegativesRankingLoss
 base_model: Snowflake/snowflake-arctic-embed-m-v1.5
 widget:
-- source_sentence: What are the abstract methods provided for managing model servers
-    in ZenML's BaseModelDeployerFlavor class?
   sentences:
-  - "quired for your GCP Image Builder by running e.g.:zenml service-connector list-resources\
-    \ --resource-type gcp-generic\n\nExample Command Output\n\nThe following 'gcp-generic'\
-    \ resources can be accessed by service connectors that you have configured:\n\
-    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┓\n\
-    ┃             CONNECTOR ID             │ CONNECTOR NAME │ CONNECTOR TYPE │ RESOURCE\
-    \ TYPE  │ RESOURCE NAMES ┃\n┠──────────────────────────────────────┼────────────────┼────────────────┼────────────────┼────────────────┨\n\
-    ┃ bfdb657d-d808-47e7-9974-9ba6e4919d83 │ gcp-generic    │ \U0001F535 gcp     \
-    \    │ \U0001F535 gcp-generic │ zenml-core     ┃\n┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┛\n\
-    \nAfter having set up or decided on a GCP Service Connector to use to authenticate\
-    \ to GCP, you can register the GCP Image Builder as follows:\n\nzenml image-builder\
-    \ register <IMAGE_BUILDER_NAME> \\\n    --flavor=gcp \\\n    --cloud_builder_image=<BUILDER_IMAGE_NAME>\
-    \ \\\n    --network=<DOCKER_NETWORK> \\\n    --build_timeout=<BUILD_TIMEOUT_IN_SECONDS>\n\
-    \n# Connect the GCP Image Builder to GCP via a GCP Service Connector\nzenml image-builder\
-    \ connect <IMAGE_BUILDER_NAME> -i\n\nA non-interactive version that connects the\
-    \ GCP Image Builder to a target GCP Service Connector:\n\nzenml image-builder\
-    \ connect <IMAGE_BUILDER_NAME> --connector <CONNECTOR_ID>\n\nExample Command Output"
   - '🧙Installation
@@ -116,6 +233,50 @@ widget:
     Running with Docker'
   - ":\n        \"\"\"Abstract method to deploy a model.\"\"\"@staticmethod\n    @abstractmethod\n\
     \    def get_model_server_info(\n            service: BaseService,\n    ) -> Dict[str,\
     \ Optional[str]]:\n        \"\"\"Give implementation-specific way to extract relevant\
@@ -143,324 +304,177 @@ widget:
     \ version of the base implementation which aims to highlight the abstraction layer.\
     \ In order to see the full implementation and get the complete docstrings, please\
     \ check the SDK docs .\n\nBuilding your own model deployers"
-- source_sentence: How can you successfully connect the image builder `gcp-image-builder`
-    to the resources using a connector ID?
   sentences:
-  - 'ZenML - Bridging the gap between ML & Ops
-    Legacy Docs
-    Bleeding EdgeLegacy Docs0.67.0
-    🧙‍♂️Find older version our docs
-    Powered by GitBook'
-  - 'Google Cloud Image Builder
-    Building container images with Google Cloud Build
-    The Google Cloud image builder is an image builder flavor provided by the ZenML
-    gcp integration that uses Google Cloud Build to build container images.
-    When to use it
-    You should use the Google Cloud image builder if:
-    you''re unable to install or use Docker on your client machine.
-    you''re already using GCP.
-    your stack is mainly composed of other Google Cloud components such as the GCS
-    Artifact Store or the Vertex Orchestrator.
-    How to deploy it
-    Would you like to skip ahead and deploy a full ZenML cloud stack already, including
-    the Google Cloud image builder? Check out the in-browser stack deployment wizard,
-    the stack registration wizard, or the ZenML GCP Terraform module for a shortcut
-    on how to deploy & register this stack component.
-    In order to use the ZenML Google Cloud image builder you need to enable Google
-    Cloud Build relevant APIs on the Google Cloud project.
-    How to use it
-    To use the Google Cloud image builder, we need:
-    The ZenML gcp integration installed. If you haven''t done so, run:
-    zenml integration install gcp
-    A GCP Artifact Store where the build context will be uploaded, so Google Cloud
-    Build can access it.
-    A GCP container registry where the built image will be pushed.
-    Optionally, the GCP project ID in which you want to run the build and a service
-    account with the needed permissions to run the build. If not provided, then the
-    project ID and credentials will be inferred from the environment.
-    Optionally, you can change:
-    the Docker image used by Google Cloud Build to execute the steps to build and
-    push the Docker image. By default, the builder image will be ''gcr.io/cloud-builders/docker''.
-    The network to which the container used to build the ZenML pipeline Docker image
-    will be attached. More information: Cloud build network.
-    The build timeout for the build, and for the blocking operation waiting for the
-    build to finish. More information: Build Timeout.'
-  - "--connector <CONNECTOR_ID>\n\nExample Command Output$ zenml image-builder connect\
-    \ gcp-image-builder --connector gcp-generic\nSuccessfully connected image builder\
-    \ `gcp-image-builder` to the following resources:\n┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┓\n\
     ┃             CONNECTOR ID             │ CONNECTOR NAME │ CONNECTOR TYPE │ RESOURCE\
     \ TYPE  │ RESOURCE NAMES ┃\n┠──────────��───────────────────────────┼────────────────┼────────────────┼────────────────┼────────────────┨\n\
     ┃ bfdb657d-d808-47e7-9974-9ba6e4919d83 │ gcp-generic    │ \U0001F535 gcp     \
     \    │ \U0001F535 gcp-generic │ zenml-core     ┃\n┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┛\n\
-    \nAs a final step, you can use the GCP Image Builder in a ZenML Stack:\n\n# Register\
-    \ and set a stack with the new image builder\nzenml stack register <STACK_NAME>\
-    \ -i <IMAGE_BUILDER_NAME> ... --set\n\nWhen you register the GCP Image Builder,\
-    \ you can generate a GCP Service Account Key, save it to a local file and then\
-    \ reference it in the Image Builder configuration.\n\nThis method has the advantage\
-    \ that you don't need to install and configure the GCP CLI on your host, but it's\
-    \ still not as secure as using a GCP Service Connector and the stack component\
-    \ configuration is not portable to other hosts.\n\nFor this method, you need to\
-    \ create a user-managed GCP service account, and grant it privileges to access\
-    \ the Cloud Build API and to run Cloud Builder jobs (e.g. the Cloud Build Editor\
-    \ IAM role.\n\nWith the service account key downloaded to a local file, you can\
-    \ register the GCP Image Builder as follows:\n\nzenml image-builder register <IMAGE_BUILDER_NAME>\
-    \ \\\n    --flavor=gcp \\\n    --project=<GCP_PROJECT_ID> \\\n    --service_account_path=<PATH_TO_SERVICE_ACCOUNT_KEY>\
-    \ \\\n    --cloud_builder_image=<BUILDER_IMAGE_NAME> \\\n    --network=<DOCKER_NETWORK>\
-    \ \\\n    --build_timeout=<BUILD_TIMEOUT_IN_SECONDS>"
-- source_sentence: How do I finetune embeddings using Sentence Transformers in ZenML?
-  sentences:
-  - "nsible for cluster-manager-specific configuration._io_configuration is a critical\
-    \ method. Even though we have materializers, Spark might require additional packages\
-    \ and configuration to work with a specific filesystem. This method is used as\
-    \ an interface to provide this configuration.\n\n_additional_configuration takes\
-    \ the submit_args, converts, and appends them to the overall configuration.\n\n\
-    Once the configuration is completed, _launch_spark_job comes into play. This takes\
-    \ the completed configuration and runs a Spark job on the given master URL with\
-    \ the specified deploy_mode. By default, this is achieved by creating and executing\
-    \ a spark-submit command.\n\nWarning\n\nIn its first iteration, the pre-configuration\
-    \ with _io_configuration method is only effective when it is paired with an S3ArtifactStore\
-    \ (which has an authentication secret). When used with other artifact store flavors,\
-    \ you might be required to provide additional configuration through the submit_args.\n\
-    \nStack Component: KubernetesSparkStepOperator\n\nThe KubernetesSparkStepOperator\
-    \ is implemented by subclassing the base SparkStepOperator and uses the PipelineDockerImageBuilder\
-    \ class to build and push the required Docker images.\n\nfrom typing import Optional\n\
-    \nfrom zenml.integrations.spark.step_operators.spark_step_operator import (\n\
-    \    SparkStepOperatorConfig\n)\n\nclass KubernetesSparkStepOperatorConfig(SparkStepOperatorConfig):\n\
-    \    \"\"\"Config for the Kubernetes Spark step operator.\"\"\"\n\nnamespace:\
-    \ Optional[str] = None\n    service_account: Optional[str] = None\n\nfrom pyspark.conf\
-    \ import SparkConf\n\nfrom zenml.utils.pipeline_docker_image_builder import PipelineDockerImageBuilder\n\
-    from zenml.integrations.spark.step_operators.spark_step_operator import (\n  \
-    \  SparkStepOperator\n)\n\nclass KubernetesSparkStepOperator(SparkStepOperator):\n\
-    \    \"\"\"Step operator which runs Steps with Spark on Kubernetes.\"\"\""
-  - 'Finetuning embeddings with Sentence Transformers
-    Finetune embeddings with Sentence Transformers.
-    PreviousSynthetic data generationNextEvaluating finetuned embeddings
-    Last updated 1 month ago'
-  - 'Whylogs
-    How to collect and visualize statistics to track changes in your pipelines'' data
-    with whylogs/WhyLabs profiling.
-    The whylogs/WhyLabs Data Validator flavor provided with the ZenML integration
-    uses whylogs and WhyLabs to generate and track data profiles, highly accurate
-    descriptive representations of your data. The profiles can be used to implement
-    automated corrective actions in your pipelines, or to render interactive representations
-    for further visual interpretation, evaluation and documentation.
-    When would you want to use it?
-    Whylogs is an open-source library that analyzes your data and creates statistical
-    summaries called whylogs profiles. Whylogs profiles can be processed in your pipelines
-    and visualized locally or uploaded to the WhyLabs platform, where more in depth
-    analysis can be carried out. Even though whylogs also supports other data types,
-    the ZenML whylogs integration currently only works with tabular data in pandas.DataFrame
-    format.
-    You should use the whylogs/WhyLabs Data Validator when you need the following
-    data validation features that are possible with whylogs and WhyLabs:
-    Data Quality: validate data quality in model inputs or in a data pipeline
-    Data Drift: detect data drift in model input features
-    Model Drift: Detect training-serving skew, concept drift, and model performance
-    degradation
-    You should consider one of the other Data Validator flavors if you need a different
-    set of data validation features.
-    How do you deploy it?
-    The whylogs Data Validator flavor is included in the whylogs ZenML integration,
-    you need to install it on your local machine to be able to register a whylogs
-    Data Validator and add it to your stack:
-    zenml integration install whylogs -y
-    If you don''t need to connect to the WhyLabs platform to upload and store the
-    generated whylogs data profiles, the Data Validator stack component does not require
-    any configuration parameters. Adding it to a stack is as simple as running e.g.:'
-- source_sentence: What is the purpose of ZenML in the context of ML and Ops?
-  sentences:
-  - "> \\\n    --build_timeout=<BUILD_TIMEOUT_IN_SECONDS># Register and set a stack\
-    \ with the new image builder\nzenml stack register <STACK_NAME> -i <IMAGE_BUILDER_NAME>\
-    \ ... --set\n\nCaveats\n\nAs described in this Google Cloud Build documentation\
-    \ page, Google Cloud Build uses containers to execute the build steps which are\
-    \ automatically attached to a network called cloudbuild that provides some Application\
-    \ Default Credentials (ADC), that allow the container to be authenticated and\
-    \ therefore use other GCP services.\n\nBy default, the GCP Image Builder is executing\
-    \ the build command of the ZenML Pipeline Docker image with the option --network=cloudbuild,\
-    \ so the ADC provided by the cloudbuild network can also be used in the build.\
-    \ This is useful if you want to install a private dependency from a GCP Artifact\
-    \ Registry, but you will also need to use a custom base parent image with the\
-    \ keyrings.google-artifactregistry-auth installed, so pip can connect and authenticate\
-    \ in the private artifact registry to download the dependency.\n\nFROM zenmldocker/zenml:latest\n\
-    \nRUN pip install keyrings.google-artifactregistry-auth\n\nThe above Dockerfile\
-    \ uses zenmldocker/zenml:latest as a base image, but is recommended to change\
-    \ the tag to specify the ZenML version and Python version like 0.33.0-py3.10.\n\
-    \nPreviousKaniko Image BuilderNextDevelop a Custom Image Builder\n\nLast updated\
-    \ 21 days ago"
-  - "Control caching behavior\n\nBy default steps in ZenML pipelines are cached whenever\
-    \ code and parameters stay unchanged.\n\n@step(enable_cache=True) # set cache\
-    \ behavior at step level\ndef load_data(parameter: int) -> dict:\n    ...\n\n\
-    @step(enable_cache=False) # settings at step level override pipeline level\ndef\
-    \ train_model(data: dict) -> None:\n    ...\n\n@pipeline(enable_cache=True) #\
-    \ set cache behavior at step level\ndef simple_ml_pipeline(parameter: int):\n\
-    \    ...\n\nCaching only happens when code and parameters stay the same.\n\nLike\
-    \ many other step and pipeline settings, you can also change this afterward:\n\
-    \n# Same as passing it in the step decorator\nmy_step.configure(enable_cache=...)\n\
-    \n# Same as passing it in the pipeline decorator\nmy_pipeline.configure(enable_cache=...)\n\
-    \nFind out here how to configure this in a YAML file\n\nPreviousStep output typing\
-    \ and annotationNextSchedule a pipeline\n\nLast updated 4 months ago"
-  - 'ZenML - Bridging the gap between ML & Ops
-    Legacy Docs
-    Bleeding EdgeLegacy Docs0.67.0
-    🧙‍♂️Find older version our docs
-    Powered by GitBook'
-- source_sentence: How does ZenML facilitate the flow of data between steps in a pipeline?
   sentences:
-  - "tainer_registry \\\n    -i local_builder \\\n    --setOnce you added the step\
-    \ operator to your active stack, you can use it to execute individual steps of\
-    \ your pipeline by specifying it in the @step decorator as follows:\n\nfrom zenml\
-    \ import step\n\n@step(step_operator=<STEP_OPERATOR_NAME>)\ndef step_on_spark(...)\
-    \ -> ...:\n    \"\"\"Some step that should run with Spark on Kubernetes.\"\"\"\
-    \n    ...\n\nAfter successfully running any step with a KubernetesSparkStepOperator,\
-    \ you should be able to see that a Spark driver pod was created in your cluster\
-    \ for each pipeline step when running kubectl get pods -n $KUBERNETES_NAMESPACE.\n\
-    \nInstead of hardcoding a step operator name, you can also use the Client to dynamically\
-    \ use the step operator of your active stack:\n\nfrom zenml.client import Client\n\
-    \nstep_operator = Client().active_stack.step_operator\n\n@step(step_operator=step_operator.name)\n\
-    def step_on_spark(...) -> ...:\n    ...\n\nAdditional configuration\n\nFor additional\
-    \ configuration of the Spark step operator, you can pass SparkStepOperatorSettings\
-    \ when defining or running your pipeline. Check out the SDK docs for a full list\
-    \ of available attributes and this docs page for more information on how to specify\
-    \ settings.\n\nPreviousKubernetesNextDevelop a Custom Step Operator\n\nLast updated\
-    \ 4 months ago"
-  - "\U0001F5C4️Handle Data/Artifacts\n\nStep outputs in ZenML are stored in the artifact\
-    \ store. This enables caching, lineage and auditability. Using type annotations\
-    \ helps with transparency, passing data between steps, and serializing/des\n\n\
-    For best results, use type annotations for your outputs. This is good coding practice\
-    \ for transparency, helps ZenML handle passing data between steps, and also enables\
-    \ ZenML to serialize and deserialize (referred to as 'materialize' in ZenML) the\
-    \ data.\n\n@step\ndef load_data(parameter: int) -> Dict[str, Any]:\n\n# do something\
-    \ with the parameter here\n\ntraining_data = [[1, 2], [3, 4], [5, 6]]\n    labels\
-    \ = [0, 1, 0]\n    return {'features': training_data, 'labels': labels}\n\n@step\n\
-    def train_model(data: Dict[str, Any]) -> None:\n    total_features = sum(map(sum,\
-    \ data['features']))\n    total_labels = sum(data['labels'])\n    \n    # Train\
-    \ some model here\n    \n    print(f\"Trained model using {len(data['features'])}\
-    \ data points. \"\n          f\"Feature sum is {total_features}, label sum is\
-    \ {total_labels}\")\n\n@pipeline  \ndef simple_ml_pipeline(parameter: int):\n\
-    \    dataset = load_data(parameter=parameter)  # Get the output \n    train_model(dataset)\
-    \  # Pipe the previous step output into the downstream step\n\nIn this code, we\
-    \ define two steps: load_data and train_model. The load_data step takes an integer\
-    \ parameter and returns a dictionary containing training data and labels. The\
-    \ train_model step receives the dictionary from load_data, extracts the features\
-    \ and labels, and trains a model (not shown here).\n\nFinally, we define a pipeline\
-    \ simple_ml_pipeline that chains the load_data and train_model steps together.\
-    \ The output from load_data is passed as input to train_model, demonstrating how\
-    \ data flows between steps in a ZenML pipeline.\n\nPreviousDisable colorful loggingNextHow\
-    \ ZenML stores data\n\nLast updated 4 months ago"
-  - "in the ZenML dashboard.\n\nThe whylogs standard stepZenML wraps the whylogs/WhyLabs\
-    \ functionality in the form of a standard WhylogsProfilerStep step. The only field\
-    \ in the step config is a dataset_timestamp attribute which is only relevant when\
-    \ you upload the profiles to WhyLabs that uses this field to group and merge together\
-    \ profiles belonging to the same dataset. The helper function get_whylogs_profiler_step\
-    \ used to create an instance of this standard step takes in an optional dataset_id\
-    \ parameter that is also used only in the context of WhyLabs upload to identify\
-    \ the model in the context of which the profile is uploaded, e.g.:\n\nfrom zenml.integrations.whylogs.steps\
-    \ import get_whylogs_profiler_step\n\ntrain_data_profiler = get_whylogs_profiler_step(dataset_id=\"\
-    model-2\")\ntest_data_profiler = get_whylogs_profiler_step(dataset_id=\"model-3\"\
-    )\n\nThe step can then be inserted into your pipeline where it can take in a pandas.DataFrame\
-    \ dataset, e.g.:\n\nfrom zenml import pipeline\n\n@pipeline\ndef data_profiling_pipeline():\n\
-    \    data, _ = data_loader()\n    train, test = data_splitter(data)\n    train_data_profiler(train)\n\
-    \    test_data_profiler(test)\n\ndata_profiling_pipeline()\n\nAs can be seen from\
-    \ the step definition , the step takes in a dataset and returns a whylogs DatasetProfileView\
-    \ object:\n\n@step\ndef whylogs_profiler_step(\n    dataset: pd.DataFrame,\n \
-    \   dataset_timestamp: Optional[datetime.datetime] = None,\n) -> DatasetProfileView:\n\
-    \    ...\n\nYou should consult the official whylogs documentation for more information\
-    \ on what you can do with the collected profiles.\n\nYou can view the complete\
-    \ list of configuration parameters in the SDK docs.\n\nThe whylogs Data Validator\n\
-    \nThe whylogs Data Validator implements the same interface as do all Data Validators,\
-    \ so this method forces you to maintain some level of compatibility with the overall\
-    \ Data Validator abstraction, which guarantees an easier migration in case you\
-    \ decide to switch to another Data Validator."
 pipeline_tag: sentence-similarity
 library_name: sentence-transformers
 metrics:
@@ -646,7 +660,7 @@ model-index:
       type: dim_64
     metrics:
     - type: cosine_accuracy@1
-      value: 0.75
       name: Cosine Accuracy@1
     - type: cosine_accuracy@3
       value: 1.0
@@ -658,7 +672,7 @@ model-index:
       value: 1.0
       name: Cosine Accuracy@10
     - type: cosine_precision@1
-      value: 0.75
       name: Cosine Precision@1
     - type: cosine_precision@3
       value: 0.3333333333333333
@@ -670,7 +684,7 @@ model-index:
       value: 0.1
       name: Cosine Precision@10
     - type: cosine_recall@1
-      value: 0.75
       name: Cosine Recall@1
     - type: cosine_recall@3
       value: 1.0
@@ -682,13 +696,13 @@ model-index:
       value: 1.0
       name: Cosine Recall@10
     - type: cosine_ndcg@10
-      value: 0.9077324383928644
       name: Cosine Ndcg@10
     - type: cosine_mrr@10
-      value: 0.875
       name: Cosine Mrr@10
     - type: cosine_map@100
-      value: 0.875
       name: Cosine Map@100
 ---
@@ -743,9 +757,9 @@ from sentence_transformers import SentenceTransformer
 model = SentenceTransformer("zenml/finetuned-snowflake-arctic-embed-m-v1.5")
 # Run inference
 sentences = [
-    'How does ZenML facilitate the flow of data between steps in a pipeline?',
-    '🗄️Handle Data/Artifacts\n\nStep outputs in ZenML are stored in the artifact store. This enables caching, lineage and auditability. Using type annotations helps with transparency, passing data between steps, and serializing/des\n\nFor best results, use type annotations for your outputs. This is good coding practice for transparency, helps ZenML handle passing data between steps, and also enables ZenML to serialize and deserialize (referred to as \'materialize\' in ZenML) the data.\n\n@step\ndef load_data(parameter: int) -> Dict[str, Any]:\n\n# do something with the parameter here\n\ntraining_data = [[1, 2], [3, 4], [5, 6]]\n    labels = [0, 1, 0]\n    return {\'features\': training_data, \'labels\': labels}\n\n@step\ndef train_model(data: Dict[str, Any]) -> None:\n    total_features = sum(map(sum, data[\'features\']))\n    total_labels = sum(data[\'labels\'])\n    \n    # Train some model here\n    \n    print(f"Trained model using {len(data[\'features\'])} data points. "\n          f"Feature sum is {total_features}, label sum is {total_labels}")\n\n@pipeline  \ndef simple_ml_pipeline(parameter: int):\n    dataset = load_data(parameter=parameter)  # Get the output \n    train_model(dataset)  # Pipe the previous step output into the downstream step\n\nIn this code, we define two steps: load_data and train_model. The load_data step takes an integer parameter and returns a dictionary containing training data and labels. The train_model step receives the dictionary from load_data, extracts the features and labels, and trains a model (not shown here).\n\nFinally, we define a pipeline simple_ml_pipeline that chains the load_data and train_model steps together. The output from load_data is passed as input to train_model, demonstrating how data flows between steps in a ZenML pipeline.\n\nPreviousDisable colorful loggingNextHow ZenML stores data\n\nLast updated 4 months ago',
-    'in the ZenML dashboard.\n\nThe whylogs standard stepZenML wraps the whylogs/WhyLabs functionality in the form of a standard WhylogsProfilerStep step. The only field in the step config is a dataset_timestamp attribute which is only relevant when you upload the profiles to WhyLabs that uses this field to group and merge together profiles belonging to the same dataset. The helper function get_whylogs_profiler_step used to create an instance of this standard step takes in an optional dataset_id parameter that is also used only in the context of WhyLabs upload to identify the model in the context of which the profile is uploaded, e.g.:\n\nfrom zenml.integrations.whylogs.steps import get_whylogs_profiler_step\n\ntrain_data_profiler = get_whylogs_profiler_step(dataset_id="model-2")\ntest_data_profiler = get_whylogs_profiler_step(dataset_id="model-3")\n\nThe step can then be inserted into your pipeline where it can take in a pandas.DataFrame dataset, e.g.:\n\nfrom zenml import pipeline\n\n@pipeline\ndef data_profiling_pipeline():\n    data, _ = data_loader()\n    train, test = data_splitter(data)\n    train_data_profiler(train)\n    test_data_profiler(test)\n\ndata_profiling_pipeline()\n\nAs can be seen from the step definition , the step takes in a dataset and returns a whylogs DatasetProfileView object:\n\n@step\ndef whylogs_profiler_step(\n    dataset: pd.DataFrame,\n    dataset_timestamp: Optional[datetime.datetime] = None,\n) -> DatasetProfileView:\n    ...\n\nYou should consult the official whylogs documentation for more information on what you can do with the collected profiles.\n\nYou can view the complete list of configuration parameters in the SDK docs.\n\nThe whylogs Data Validator\n\nThe whylogs Data Validator implements the same interface as do all Data Validators, so this method forces you to maintain some level of compatibility with the overall Data Validator abstraction, which guarantees an easier migration in case you decide to switch to another Data Validator.',
 ]
 embeddings = model.encode(sentences)
 print(embeddings.shape)
@@ -855,23 +869,23 @@ You can finetune this model on your own dataset.
 * Dataset: `dim_64`
 * Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
-| Metric              | Value     |
-|:--------------------|:----------|
-| cosine_accuracy@1   | 0.75      |
-| cosine_accuracy@3   | 1.0       |
-| cosine_accuracy@5   | 1.0       |
-| cosine_accuracy@10  | 1.0       |
-| cosine_precision@1  | 0.75      |
-| cosine_precision@3  | 0.3333    |
-| cosine_precision@5  | 0.2       |
-| cosine_precision@10 | 0.1       |
-| cosine_recall@1     | 0.75      |
-| cosine_recall@3     | 1.0       |
-| cosine_recall@5     | 1.0       |
-| cosine_recall@10    | 1.0       |
-| cosine_ndcg@10      | 0.9077    |
-| cosine_mrr@10       | 0.875     |
-| **cosine_map@100**  | **0.875** |
 <!--
 ## Bias, Risks and Limitations
@@ -898,13 +912,13 @@ You can finetune this model on your own dataset.
   |         | positive                                                                           | anchor                                                                               |
   |:--------|:-----------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|
   | type    | string                                                                             | string                                                                               |
-  | details | <ul><li>min: 13 tokens</li><li>mean: 23.92 tokens</li><li>max: 41 tokens</li></ul> | <ul><li>min: 31 tokens</li><li>mean: 321.11 tokens</li><li>max: 512 tokens</li></ul> |
 * Samples:
-  | positive                                                                                                                            | anchor                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
-  |:------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-  | <code>How does ZenML integrate Spark step operators for executing individual steps?</code>                                          | <code>Spark<br><br>Executing individual steps on Spark<br><br>The spark integration brings two different step operators:<br><br>Step Operator: The SparkStepOperator serves as the base class for all the Spark-related step operators.<br><br>Step Operator: The KubernetesSparkStepOperator is responsible for launching ZenML steps as Spark applications with Kubernetes as a cluster manager.<br><br>Step Operators: SparkStepOperator<br><br>A summarized version of the implementation can be summarized in two parts. First, the configuration:<br><br>from typing import Optional, Dict, Any<br>from zenml.step_operators import BaseStepOperatorConfig<br><br>class SparkStepOperatorConfig(BaseStepOperatorConfig):<br>    """Spark step operator config.<br><br>Attributes:<br>        master: is the master URL for the cluster. You might see different<br>            schemes for different cluster managers which are supported by Spark<br>            like Mesos, YARN, or Kubernetes. Within the context of this PR,<br>            the implementation supports Kubernetes as a cluster manager.<br>        deploy_mode: can either be 'cluster' (default) or 'client' and it<br>            decides where the driver node of the application will run.<br>        submit_kwargs: is the JSON string of a dict, which will be used<br>            to define additional params if required (Spark has quite a<br>            lot of different parameters, so including them, all in the step<br>            operator was not implemented).<br>    """<br><br>master: str<br>    deploy_mode: str = "cluster"<br>    submit_kwargs: Optional[Dict[str, Any]] = None<br><br>and then the implementation:<br><br>from typing import List<br>from pyspark.conf import SparkConf<br><br>from zenml.step_operators import BaseStepOperator<br><br>class SparkStepOperator(BaseStepOperator):<br>    """Base class for all Spark-related step operators."""<br><br>def _resource_configuration(<br>            self,<br>            spark_config: SparkConf,<br>            resource_configuration: "ResourceSettings",<br>    ) -> None:<br>        """Configures Spark to handle the resource configuration."""</code> |
-  | <code>How can ZenML be used to finetune LLMs for specific tasks or to improve performance and cost?</code>                          | <code>Finetuning LLMs with ZenML<br><br>Finetune LLMs for specific tasks or to improve performance and cost.<br><br>PreviousEvaluating finetuned embeddingsNextSet up a project repository<br><br>Last updated 6 months ago</code>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
-  | <code>How can I develop a custom model deployer in ZenML for efficient deployment and management of machine-learning models?</code> | <code>Develop a Custom Model Deployer<br><br>Learning how to develop a custom model deployer.<br><br>Before diving into the specifics of this component type, it is beneficial to familiarize yourself with our general guide to writing custom component flavors in ZenML. This guide provides an essential understanding of ZenML's component flavor concepts.<br><br>To deploy and manage your trained machine-learning models, ZenML provides a stack component called Model Deployer. This component is responsible for interacting with the deployment tool, framework, or platform.<br><br>When present in a stack, the model deployer can also act as a registry for models that are served with ZenML. You can use the model deployer to list all models that are currently deployed for online inference or filtered according to a particular pipeline run or step, or to suspend, resume or delete an external model server managed through ZenML.<br><br>Base Abstraction<br><br>In ZenML, the base abstraction of the model deployer is built on top of three major criteria:<br><br>It needs to ensure efficient deployment and management of models in accordance with the specific requirements of the serving infrastructure, by holding all the stack-related configuration attributes required to interact with the remote model serving tool, service, or platform.<br><br>It needs to implement the continuous deployment logic necessary to deploy models in a way that updates an existing model server that is already serving a previous version of the same model instead of creating a new model server for every new model version (see the deploy_model abstract method). This functionality can be consumed directly from ZenML pipeline steps, but it can also be used outside the pipeline to deploy ad-hoc models. It is also usually coupled with a standard model deployer step, implemented by each integration, that hides the details of the deployment process from the user.</code>                                                                                                                                                                                                        |
 * Loss: [<code>MatryoshkaLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters:
   ```json
   {
@@ -1061,9 +1075,9 @@ You can finetune this model on your own dataset.
 ### Training Logs
 | Epoch   | Step  | dim_384_cosine_map@100 | dim_256_cosine_map@100 | dim_128_cosine_map@100 | dim_64_cosine_map@100 |
 |:-------:|:-----:|:----------------------:|:----------------------:|:----------------------:|:---------------------:|
-| 1.0     | 1     | 1.0                    | 1.0                    | 0.875                  | 0.875                 |
-| **2.0** | **3** | **1.0**                | **1.0**                | **1.0**                | **0.875**             |
-| 3.0     | 4     | 1.0                    | 1.0                    | 1.0                    | 0.875                 |
 * The bold row denotes the saved checkpoint.

 - loss:MultipleNegativesRankingLoss
 base_model: Snowflake/snowflake-arctic-embed-m-v1.5
 widget:
+- source_sentence: How do you configure the necessary RBAC resources in Kubernetes
+    to enable Spark access for managing driver executor pods, and what are the subsequent
+    steps needed to register the stack component using ZenML?
   sentences:
+  - 'Google Cloud Image Builder
+    Building container images with Google Cloud Build
+    The Google Cloud image builder is an image builder flavor provided by the ZenML
+    gcp integration that uses Google Cloud Build to build container images.
+    When to use it
+    You should use the Google Cloud image builder if:
+    you''re unable to install or use Docker on your client machine.
+    you''re already using GCP.
+    your stack is mainly composed of other Google Cloud components such as the GCS
+    Artifact Store or the Vertex Orchestrator.
+    How to deploy it
+    Would you like to skip ahead and deploy a full ZenML cloud stack already, including
+    the Google Cloud image builder? Check out the in-browser stack deployment wizard,
+    the stack registration wizard, or the ZenML GCP Terraform module for a shortcut
+    on how to deploy & register this stack component.
+    In order to use the ZenML Google Cloud image builder you need to enable Google
+    Cloud Build relevant APIs on the Google Cloud project.
+    How to use it
+    To use the Google Cloud image builder, we need:
+    The ZenML gcp integration installed. If you haven''t done so, run:
+    zenml integration install gcp
+    A GCP Artifact Store where the build context will be uploaded, so Google Cloud
+    Build can access it.
+    A GCP container registry where the built image will be pushed.
+    Optionally, the GCP project ID in which you want to run the build and a service
+    account with the needed permissions to run the build. If not provided, then the
+    project ID and credentials will be inferred from the environment.
+    Optionally, you can change:
+    the Docker image used by Google Cloud Build to execute the steps to build and
+    push the Docker image. By default, the builder image will be ''gcr.io/cloud-builders/docker''.
+    The network to which the container used to build the ZenML pipeline Docker image
+    will be attached. More information: Cloud build network.
+    The build timeout for the build, and for the blocking operation waiting for the
+    build to finish. More information: Build Timeout.'
+  - "_run.steps[step_name]\n    whylogs_step.visualize()if __name__ == \"__main__\"\
+    :\n    visualize_statistics(\"data_loader\")\n    visualize_statistics(\"train_data_profiler\"\
+    , \"test_data_profiler\")\n\nPreviousEvidentlyNextDevelop a custom data validator\n\
+    \nLast updated 1 month ago"
+  - "ngs/python/Dockerfile -u 0 build\n\nConfiguring RBACAdditionally, you may need\
+    \ to create the several resources in Kubernetes in order to give Spark access\
+    \ to edit/manage your driver executor pods.\n\nTo do so, create a file called\
+    \ rbac.yaml with the following content:\n\napiVersion: v1\nkind: Namespace\nmetadata:\n\
+    \  name: spark-namespace\n---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n\
+    \  name: spark-service-account\n  namespace: spark-namespace\n---\napiVersion:\
+    \ rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\nmetadata:\n  name: spark-role\n\
+    \  namespace: spark-namespace\nsubjects:\n  - kind: ServiceAccount\n    name:\
+    \ spark-service-account\n    namespace: spark-namespace\nroleRef:\n  kind: ClusterRole\n\
+    \  name: edit\n  apiGroup: rbac.authorization.k8s.io\n---\n\nAnd then execute\
+    \ the following command to create the resources:\n\naws eks --region=$REGION update-kubeconfig\
+    \ --name=$EKS_CLUSTER_NAME\n\nkubectl create -f rbac.yaml\n\nLastly, note down\
+    \ the namespace and the name of the service account since you will need them when\
+    \ registering the stack component in the next step.\n\nHow to use it\n\nTo use\
+    \ the KubernetesSparkStepOperator, you need:\n\nthe ZenML spark integration. If\
+    \ you haven't installed it already, run\n\nzenml integration install spark\n\n\
+    Docker installed and running.\n\nA remote artifact store as part of your stack.\n\
+    \nA remote container registry as part of your stack.\n\nA Kubernetes cluster deployed.\n\
+    \nWe can then register the step operator and use it in our active stack:\n\nzenml\
+    \ step-operator register spark_step_operator \\\n\t--flavor=spark-kubernetes \\\
+    \n\t--master=k8s://$EKS_API_SERVER_ENDPOINT \\\n\t--namespace=<SPARK_KUBERNETES_NAMESPACE>\
+    \ \\\n\t--service_account=<SPARK_KUBERNETES_SERVICE_ACCOUNT>\n\n# Register the\
+    \ stack\nzenml stack register spark_stack \\\n    -o default \\\n    -s spark_step_operator\
+    \ \\\n    -a spark_artifact_store \\\n    -c spark_container_registry \\\n   \
+    \ -i local_builder \\\n    --set"
+- source_sentence: What is the function of a ZenML BaseService registry in the context
+    of model deployment?
+  sentences:
+  - "\U0001F5C4️Handle Data/Artifacts\n\nStep outputs in ZenML are stored in the artifact\
+    \ store. This enables caching, lineage and auditability. Using type annotations\
+    \ helps with transparency, passing data between steps, and serializing/des\n\n\
+    For best results, use type annotations for your outputs. This is good coding practice\
+    \ for transparency, helps ZenML handle passing data between steps, and also enables\
+    \ ZenML to serialize and deserialize (referred to as 'materialize' in ZenML) the\
+    \ data.\n\n@step\ndef load_data(parameter: int) -> Dict[str, Any]:\n\n# do something\
+    \ with the parameter here\n\ntraining_data = [[1, 2], [3, 4], [5, 6]]\n    labels\
+    \ = [0, 1, 0]\n    return {'features': training_data, 'labels': labels}\n\n@step\n\
+    def train_model(data: Dict[str, Any]) -> None:\n    total_features = sum(map(sum,\
+    \ data['features']))\n    total_labels = sum(data['labels'])\n    \n    # Train\
+    \ some model here\n    \n    print(f\"Trained model using {len(data['features'])}\
+    \ data points. \"\n          f\"Feature sum is {total_features}, label sum is\
+    \ {total_labels}\")\n\n@pipeline  \ndef simple_ml_pipeline(parameter: int):\n\
+    \    dataset = load_data(parameter=parameter)  # Get the output \n    train_model(dataset)\
+    \  # Pipe the previous step output into the downstream step\n\nIn this code, we\
+    \ define two steps: load_data and train_model. The load_data step takes an integer\
+    \ parameter and returns a dictionary containing training data and labels. The\
+    \ train_model step receives the dictionary from load_data, extracts the features\
+    \ and labels, and trains a model (not shown here).\n\nFinally, we define a pipeline\
+    \ simple_ml_pipeline that chains the load_data and train_model steps together.\
+    \ The output from load_data is passed as input to train_model, demonstrating how\
+    \ data flows between steps in a ZenML pipeline.\n\nPreviousDisable colorful loggingNextHow\
+    \ ZenML stores data\n\nLast updated 4 months ago"
   - '🧙Installation
     Running with Docker'
+  - "e details of the deployment process from the user.It needs to act as a ZenML\
+    \ BaseService registry, where every BaseService instance is used as an internal\
+    \ representation of a remote model server (see the find_model_server abstract\
+    \ method). To achieve this, it must be able to re-create the configuration of\
+    \ a BaseService from information that is persisted externally, alongside, or even\
+    \ as part of the remote model server configuration itself. For example, for model\
+    \ servers that are implemented as Kubernetes resources, the BaseService instances\
+    \ can be serialized and saved as Kubernetes resource annotations. This allows\
+    \ the model deployer to keep track of all externally running model servers and\
+    \ to re-create their corresponding BaseService instance representations at any\
+    \ given time. The model deployer also defines methods that implement basic life-cycle\
+    \ management on remote model servers outside the coverage of a pipeline (see stop_model_server\
+    \ , start_model_server and delete_model_server).\n\nPutting all these considerations\
+    \ together, we end up with the following interface:\n\nfrom abc import ABC, abstractmethod\n\
+    from typing import Dict, List, Optional, Type\nfrom uuid import UUID\n\nfrom zenml.enums\
+    \ import StackComponentType\nfrom zenml.services import BaseService, ServiceConfig\n\
+    from zenml.stack import StackComponent, StackComponentConfig, Flavor\n\nDEFAULT_DEPLOYMENT_START_STOP_TIMEOUT\
+    \ = 300\n\nclass BaseModelDeployerConfig(StackComponentConfig):\n    \"\"\"Base\
+    \ class for all ZenML model deployer configurations.\"\"\"\n\nclass BaseModelDeployer(StackComponent,\
+    \ ABC):\n    \"\"\"Base class for all ZenML model deployers.\"\"\"\n\n@abstractmethod\n\
+    \    def perform_deploy_model(\n        self,\n        id: UUID,\n        config:\
+    \ ServiceConfig,\n        timeout: int = DEFAULT_DEPLOYMENT_START_STOP_TIMEOUT,\n\
+    \    ) -> BaseService:\n        \"\"\"Abstract method to deploy a model.\"\"\""
+- source_sentence: How can I implement the abstract method to deploy a model using
+    ZenML?
+  sentences:
+  - "> \\\n    --build_timeout=<BUILD_TIMEOUT_IN_SECONDS># Register and set a stack\
+    \ with the new image builder\nzenml stack register <STACK_NAME> -i <IMAGE_BUILDER_NAME>\
+    \ ... --set\n\nCaveats\n\nAs described in this Google Cloud Build documentation\
+    \ page, Google Cloud Build uses containers to execute the build steps which are\
+    \ automatically attached to a network called cloudbuild that provides some Application\
+    \ Default Credentials (ADC), that allow the container to be authenticated and\
+    \ therefore use other GCP services.\n\nBy default, the GCP Image Builder is executing\
+    \ the build command of the ZenML Pipeline Docker image with the option --network=cloudbuild,\
+    \ so the ADC provided by the cloudbuild network can also be used in the build.\
+    \ This is useful if you want to install a private dependency from a GCP Artifact\
+    \ Registry, but you will also need to use a custom base parent image with the\
+    \ keyrings.google-artifactregistry-auth installed, so pip can connect and authenticate\
+    \ in the private artifact registry to download the dependency.\n\nFROM zenmldocker/zenml:latest\n\
+    \nRUN pip install keyrings.google-artifactregistry-auth\n\nThe above Dockerfile\
+    \ uses zenmldocker/zenml:latest as a base image, but is recommended to change\
+    \ the tag to specify the ZenML version and Python version like 0.33.0-py3.10.\n\
+    \nPreviousKaniko Image BuilderNextDevelop a Custom Image Builder\n\nLast updated\
+    \ 21 days ago"
   - ":\n        \"\"\"Abstract method to deploy a model.\"\"\"@staticmethod\n    @abstractmethod\n\
     \    def get_model_server_info(\n            service: BaseService,\n    ) -> Dict[str,\
     \ Optional[str]]:\n        \"\"\"Give implementation-specific way to extract relevant\
     \ version of the base implementation which aims to highlight the abstraction layer.\
     \ In order to see the full implementation and get the complete docstrings, please\
     \ check the SDK docs .\n\nBuilding your own model deployers"
+  - "se you decide to switch to another Data Validator.All you have to do is call\
+    \ the whylogs Data Validator methods when you need to interact with whylogs to\
+    \ generate data profiles. You may optionally enable whylabs logging to automatically\
+    \ upload the returned whylogs profile to WhyLabs, e.g.:\n\nimport pandas as pd\n\
+    from whylogs.core import DatasetProfileView\nfrom zenml.integrations.whylogs.data_validators.whylogs_data_validator\
+    \ import (\n    WhylogsDataValidator,\n)\nfrom zenml.integrations.whylogs.flavors.whylogs_data_validator_flavor\
+    \ import (\n    WhylogsDataValidatorSettings,\n)\nfrom zenml import step\n\nwhylogs_settings\
+    \ = WhylogsDataValidatorSettings(\n    enable_whylabs=True, dataset_id=\"<WHYLABS_DATASET_ID>\"\
+    \n)\n\n@step(\n    settings={\n        \"data_validator\": whylogs_settings\n\
+    \    }\n)\ndef data_profiler(\n        dataset: pd.DataFrame,\n) -> DatasetProfileView:\n\
+    \    \"\"\"Custom data profiler step with whylogs\n\nArgs:\n        dataset: a\
+    \ Pandas DataFrame\n\nReturns:\n        Whylogs profile generated for the data\n\
+    \    \"\"\"\n\n# validation pre-processing (e.g. dataset preparation) can take\
+    \ place here\n\ndata_validator = WhylogsDataValidator.get_active_data_validator()\n\
+    \    profile = data_validator.data_profiling(\n        dataset,\n    )\n    #\
+    \ optionally upload the profile to WhyLabs, if WhyLabs credentials are configured\n\
+    \    data_validator.upload_profile_view(profile)\n\n# validation post-processing\
+    \ (e.g. interpret results, take actions) can happen here\n\nreturn profile\n\n\
+    Have a look at the complete list of methods and parameters available in the WhylogsDataValidator\
+    \ API in the SDK docs.\n\nCall whylogs directly\n\nYou can use the whylogs library\
+    \ directly in your custom pipeline steps, and only leverage ZenML's capability\
+    \ of serializing, versioning and storing the DatasetProfileView objects in its\
+    \ Artifact Store. You may optionally enable whylabs logging to automatically upload\
+    \ the returned whylogs profile to WhyLabs, e.g.:"
+- source_sentence: How can I register and configure a GCP Service Connector for accessing
+    GCP Cloud Build services in ZenML?
   sentences:
+  - 'System Architectures
+    Different variations of the ZenML architecture depending on your needs.
+    PreviousZenML ProNextZenML SaaS
+    Last updated 21 days ago'
+  - "quired for your GCP Image Builder by running e.g.:zenml service-connector list-resources\
+    \ --resource-type gcp-generic\n\nExample Command Output\n\nThe following 'gcp-generic'\
+    \ resources can be accessed by service connectors that you have configured:\n\
+    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┓\n\
     ┃             CONNECTOR ID             │ CONNECTOR NAME │ CONNECTOR TYPE │ RESOURCE\
     \ TYPE  │ RESOURCE NAMES ┃\n┠──────────��───────────────────────────┼────────────────┼────────────────┼────────────────┼────────────────┨\n\
     ┃ bfdb657d-d808-47e7-9974-9ba6e4919d83 │ gcp-generic    │ \U0001F535 gcp     \
     \    │ \U0001F535 gcp-generic │ zenml-core     ┃\n┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┛\n\
+    \nAfter having set up or decided on a GCP Service Connector to use to authenticate\
+    \ to GCP, you can register the GCP Image Builder as follows:\n\nzenml image-builder\
+    \ register <IMAGE_BUILDER_NAME> \\\n    --flavor=gcp \\\n    --cloud_builder_image=<BUILDER_IMAGE_NAME>\
+    \ \\\n    --network=<DOCKER_NETWORK> \\\n    --build_timeout=<BUILD_TIMEOUT_IN_SECONDS>\n\
+    \n# Connect the GCP Image Builder to GCP via a GCP Service Connector\nzenml image-builder\
+    \ connect <IMAGE_BUILDER_NAME> -i\n\nA non-interactive version that connects the\
+    \ GCP Image Builder to a target GCP Service Connector:\n\nzenml image-builder\
+    \ connect <IMAGE_BUILDER_NAME> --connector <CONNECTOR_ID>\n\nExample Command Output"
+  - ' your GCP Image Builder to the GCP cloud platform.To set up the GCP Image Builder
+    to authenticate to GCP and access the GCP Cloud Build services, it is recommended
+    to leverage the many features provided by the GCP Service Connector such as auto-configuration,
+    best security practices regarding long-lived credentials and reusing the same
+    credentials across multiple stack components.
+    If you don''t already have a GCP Service Connector configured in your ZenML deployment,
+    you can register one using the interactive CLI command. You also have the option
+    to configure a GCP Service Connector that can be used to access more than just
+    the GCP Cloud Build service:
+    zenml service-connector register --type gcp -i
+    A non-interactive CLI example that leverages the Google Cloud CLI configuration
+    on your local machine to auto-configure a GCP Service Connector for the GCP Cloud
+    Build service:
+    zenml service-connector register <CONNECTOR_NAME> --type gcp --resource-type gcp-generic
+    --resource-name <GCS_BUCKET_NAME> --auto-configure
+    Example Command Output
+    $ zenml service-connector register gcp-generic --type gcp --resource-type gcp-generic
+    --auto-configure
+    Successfully registered service connector `gcp-generic` with access to the following
+    resources:
+    ┏━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┓
+    ┃ RESOURCE TYPE  │ RESOURCE NAMES ┃
+    ┠────────────────┼────────────────┨
+    ┃ 🔵 gcp-generic │ zenml-core     ┃
+    ┗━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┛
+    Note: Please remember to grant the entity associated with your GCP credentials
+    permissions to access the Cloud Build API and to run Cloud Builder jobs (e.g.
+    the Cloud Build Editor IAM role). The GCP Service Connector supports many different
+    authentication methods with different levels of security and convenience. You
+    should pick the one that best fits your use case.
+    If you already have one or more GCP Service Connectors configured in your ZenML
+    deployment, you can check which of them can be used to access generic GCP resources
+    like the GCP Image Builder required for your GCP Image Builder by running e.g.:'
+- source_sentence: How can ZenML be used to finetune LLMs for specific tasks or to
+    improve their performance and cost?
   sentences:
+  - " build to finish. More information: Build Timeout.We can register the image builder\
+    \ and use it in our active stack:\n\nzenml image-builder register <IMAGE_BUILDER_NAME>\
+    \ \\\n    --flavor=gcp \\\n    --cloud_builder_image=<BUILDER_IMAGE_NAME> \\\n\
+    \    --network=<DOCKER_NETWORK> \\\n    --build_timeout=<BUILD_TIMEOUT_IN_SECONDS>\n\
+    \n# Register and activate a stack with the new image builder\nzenml stack register\
+    \ <STACK_NAME> -i <IMAGE_BUILDER_NAME> ... --set\n\nYou also need to set up authentication\
+    \ required to access the Cloud Build GCP services.\n\nAuthentication Methods\n\
+    \nIntegrating and using a GCP Image Builder in your pipelines is not possible\
+    \ without employing some form of authentication. If you're looking for a quick\
+    \ way to get started locally, you can use the Local Authentication method. However,\
+    \ the recommended way to authenticate to the GCP cloud platform is through a GCP\
+    \ Service Connector. This is particularly useful if you are configuring ZenML\
+    \ stacks that combine the GCP Image Builder with other remote stack components\
+    \ also running in GCP.\n\nThis method uses the implicit GCP authentication available\
+    \ in the environment where the ZenML code is running. On your local machine, this\
+    \ is the quickest way to configure a GCP Image Builder. You don't need to supply\
+    \ credentials explicitly when you register the GCP Image Builder, as it leverages\
+    \ the local credentials and configuration that the Google Cloud CLI stores on\
+    \ your local machine. However, you will need to install and set up the Google\
+    \ Cloud CLI on your machine as a prerequisite, as covered in the Google Cloud\
+    \ documentation , before you register the GCP Image Builder.\n\nStacks using the\
+    \ GCP Image Builder set up with local authentication are not portable across environments.\
+    \ To make ZenML pipelines fully portable, it is recommended to use a GCP Service\
+    \ Connector to authenticate your GCP Image Builder to the GCP cloud platform."
+  - 'Finetuning LLMs with ZenML
+    Finetune LLMs for specific tasks or to improve performance and cost.
+    PreviousEvaluating finetuned embeddingsNextSet up a project repository
+    Last updated 6 months ago'
+  - "Spark\n\nExecuting individual steps on Spark\n\nThe spark integration brings\
+    \ two different step operators:\n\nStep Operator: The SparkStepOperator serves\
+    \ as the base class for all the Spark-related step operators.\n\nStep Operator:\
+    \ The KubernetesSparkStepOperator is responsible for launching ZenML steps as\
+    \ Spark applications with Kubernetes as a cluster manager.\n\nStep Operators:\
+    \ SparkStepOperator\n\nA summarized version of the implementation can be summarized\
+    \ in two parts. First, the configuration:\n\nfrom typing import Optional, Dict,\
+    \ Any\nfrom zenml.step_operators import BaseStepOperatorConfig\n\nclass SparkStepOperatorConfig(BaseStepOperatorConfig):\n\
+    \    \"\"\"Spark step operator config.\n\nAttributes:\n        master: is the\
+    \ master URL for the cluster. You might see different\n            schemes for\
+    \ different cluster managers which are supported by Spark\n            like Mesos,\
+    \ YARN, or Kubernetes. Within the context of this PR,\n            the implementation\
+    \ supports Kubernetes as a cluster manager.\n        deploy_mode: can either be\
+    \ 'cluster' (default) or 'client' and it\n            decides where the driver\
+    \ node of the application will run.\n        submit_kwargs: is the JSON string\
+    \ of a dict, which will be used\n            to define additional params if required\
+    \ (Spark has quite a\n            lot of different parameters, so including them,\
+    \ all in the step\n            operator was not implemented).\n    \"\"\"\n\n\
+    master: str\n    deploy_mode: str = \"cluster\"\n    submit_kwargs: Optional[Dict[str,\
+    \ Any]] = None\n\nand then the implementation:\n\nfrom typing import List\nfrom\
+    \ pyspark.conf import SparkConf\n\nfrom zenml.step_operators import BaseStepOperator\n\
+    \nclass SparkStepOperator(BaseStepOperator):\n    \"\"\"Base class for all Spark-related\
+    \ step operators.\"\"\"\n\ndef _resource_configuration(\n            self,\n \
+    \           spark_config: SparkConf,\n            resource_configuration: \"ResourceSettings\"\
+    ,\n    ) -> None:\n        \"\"\"Configures Spark to handle the resource configuration.\"\
+    \"\""
 pipeline_tag: sentence-similarity
 library_name: sentence-transformers
 metrics:
       type: dim_64
     metrics:
     - type: cosine_accuracy@1
+      value: 1.0
       name: Cosine Accuracy@1
     - type: cosine_accuracy@3
       value: 1.0
       value: 1.0
       name: Cosine Accuracy@10
     - type: cosine_precision@1
+      value: 1.0
       name: Cosine Precision@1
     - type: cosine_precision@3
       value: 0.3333333333333333
       value: 0.1
       name: Cosine Precision@10
     - type: cosine_recall@1
+      value: 1.0
       name: Cosine Recall@1
     - type: cosine_recall@3
       value: 1.0
       value: 1.0
       name: Cosine Recall@10
     - type: cosine_ndcg@10
+      value: 1.0
       name: Cosine Ndcg@10
     - type: cosine_mrr@10
+      value: 1.0
       name: Cosine Mrr@10
     - type: cosine_map@100
+      value: 1.0
       name: Cosine Map@100
 ---
 model = SentenceTransformer("zenml/finetuned-snowflake-arctic-embed-m-v1.5")
 # Run inference
 sentences = [
+    'How can ZenML be used to finetune LLMs for specific tasks or to improve their performance and cost?',
+    'Finetuning LLMs with ZenML\n\nFinetune LLMs for specific tasks or to improve performance and cost.\n\nPreviousEvaluating finetuned embeddingsNextSet up a project repository\n\nLast updated 6 months ago',
+    'Spark\n\nExecuting individual steps on Spark\n\nThe spark integration brings two different step operators:\n\nStep Operator: The SparkStepOperator serves as the base class for all the Spark-related step operators.\n\nStep Operator: The KubernetesSparkStepOperator is responsible for launching ZenML steps as Spark applications with Kubernetes as a cluster manager.\n\nStep Operators: SparkStepOperator\n\nA summarized version of the implementation can be summarized in two parts. First, the configuration:\n\nfrom typing import Optional, Dict, Any\nfrom zenml.step_operators import BaseStepOperatorConfig\n\nclass SparkStepOperatorConfig(BaseStepOperatorConfig):\n    """Spark step operator config.\n\nAttributes:\n        master: is the master URL for the cluster. You might see different\n            schemes for different cluster managers which are supported by Spark\n            like Mesos, YARN, or Kubernetes. Within the context of this PR,\n            the implementation supports Kubernetes as a cluster manager.\n        deploy_mode: can either be \'cluster\' (default) or \'client\' and it\n            decides where the driver node of the application will run.\n        submit_kwargs: is the JSON string of a dict, which will be used\n            to define additional params if required (Spark has quite a\n            lot of different parameters, so including them, all in the step\n            operator was not implemented).\n    """\n\nmaster: str\n    deploy_mode: str = "cluster"\n    submit_kwargs: Optional[Dict[str, Any]] = None\n\nand then the implementation:\n\nfrom typing import List\nfrom pyspark.conf import SparkConf\n\nfrom zenml.step_operators import BaseStepOperator\n\nclass SparkStepOperator(BaseStepOperator):\n    """Base class for all Spark-related step operators."""\n\ndef _resource_configuration(\n            self,\n            spark_config: SparkConf,\n            resource_configuration: "ResourceSettings",\n    ) -> None:\n        """Configures Spark to handle the resource configuration."""',
 ]
 embeddings = model.encode(sentences)
 print(embeddings.shape)
 * Dataset: `dim_64`
 * Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
+| Metric              | Value   |
+|:--------------------|:--------|
+| cosine_accuracy@1   | 1.0     |
+| cosine_accuracy@3   | 1.0     |
+| cosine_accuracy@5   | 1.0     |
+| cosine_accuracy@10  | 1.0     |
+| cosine_precision@1  | 1.0     |
+| cosine_precision@3  | 0.3333  |
+| cosine_precision@5  | 0.2     |
+| cosine_precision@10 | 0.1     |
+| cosine_recall@1     | 1.0     |
+| cosine_recall@3     | 1.0     |
+| cosine_recall@5     | 1.0     |
+| cosine_recall@10    | 1.0     |
+| cosine_ndcg@10      | 1.0     |
+| cosine_mrr@10       | 1.0     |
+| **cosine_map@100**  | **1.0** |
 <!--
 ## Bias, Risks and Limitations
   |         | positive                                                                           | anchor                                                                               |
   |:--------|:-----------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|
   | type    | string                                                                             | string                                                                               |
+  | details | <ul><li>min: 13 tokens</li><li>mean: 23.19 tokens</li><li>max: 48 tokens</li></ul> | <ul><li>min: 31 tokens</li><li>mean: 320.53 tokens</li><li>max: 512 tokens</li></ul> |
 * Samples:
+  | positive                                                                                                                      | anchor                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
+  |:------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+  | <code>Where can I find older versions of the ZenML documentation?</code>                                                      | <code>ZenML - Bridging the gap between ML & Ops<br><br>Legacy Docs<br><br>Bleeding EdgeLegacy Docs0.67.0<br><br>🧙‍♂️Find older version our docs<br><br>Powered by GitBook</code>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
+  | <code>How can I set up authentication for a GCP Image Builder when registering it in ZenML?</code>                            | <code> build to finish. More information: Build Timeout.We can register the image builder and use it in our active stack:<br><br>zenml image-builder register <IMAGE_BUILDER_NAME> \<br>    --flavor=gcp \<br>    --cloud_builder_image=<BUILDER_IMAGE_NAME> \<br>    --network=<DOCKER_NETWORK> \<br>    --build_timeout=<BUILD_TIMEOUT_IN_SECONDS><br><br># Register and activate a stack with the new image builder<br>zenml stack register <STACK_NAME> -i <IMAGE_BUILDER_NAME> ... --set<br><br>You also need to set up authentication required to access the Cloud Build GCP services.<br><br>Authentication Methods<br><br>Integrating and using a GCP Image Builder in your pipelines is not possible without employing some form of authentication. If you're looking for a quick way to get started locally, you can use the Local Authentication method. However, the recommended way to authenticate to the GCP cloud platform is through a GCP Service Connector. This is particularly useful if you are configuring ZenML stacks that combine the GCP Image Builder with other remote stack components also running in GCP.<br><br>This method uses the implicit GCP authentication available in the environment where the ZenML code is running. On your local machine, this is the quickest way to configure a GCP Image Builder. You don't need to supply credentials explicitly when you register the GCP Image Builder, as it leverages the local credentials and configuration that the Google Cloud CLI stores on your local machine. However, you will need to install and set up the Google Cloud CLI on your machine as a prerequisite, as covered in the Google Cloud documentation , before you register the GCP Image Builder.<br><br>Stacks using the GCP Image Builder set up with local authentication are not portable across environments. To make ZenML pipelines fully portable, it is recommended to use a GCP Service Connector to authenticate your GCP Image Builder to the GCP cloud platform.</code>                                                                                                                       |
+  | <code>How can I switch to another Data Validator and enable WhyLabs logging for automatic profile uploads using ZenML?</code> | <code>se you decide to switch to another Data Validator.All you have to do is call the whylogs Data Validator methods when you need to interact with whylogs to generate data profiles. You may optionally enable whylabs logging to automatically upload the returned whylogs profile to WhyLabs, e.g.:<br><br>import pandas as pd<br>from whylogs.core import DatasetProfileView<br>from zenml.integrations.whylogs.data_validators.whylogs_data_validator import (<br>    WhylogsDataValidator,<br>)<br>from zenml.integrations.whylogs.flavors.whylogs_data_validator_flavor import (<br>    WhylogsDataValidatorSettings,<br>)<br>from zenml import step<br><br>whylogs_settings = WhylogsDataValidatorSettings(<br>    enable_whylabs=True, dataset_id="<WHYLABS_DATASET_ID>"<br>)<br><br>@step(<br>    settings={<br>        "data_validator": whylogs_settings<br>    }<br>)<br>def data_profiler(<br>        dataset: pd.DataFrame,<br>) -> DatasetProfileView:<br>    """Custom data profiler step with whylogs<br><br>Args:<br>        dataset: a Pandas DataFrame<br><br>Returns:<br>        Whylogs profile generated for the data<br>    """<br><br># validation pre-processing (e.g. dataset preparation) can take place here<br><br>data_validator = WhylogsDataValidator.get_active_data_validator()<br>    profile = data_validator.data_profiling(<br>        dataset,<br>    )<br>    # optionally upload the profile to WhyLabs, if WhyLabs credentials are configured<br>    data_validator.upload_profile_view(profile)<br><br># validation post-processing (e.g. interpret results, take actions) can happen here<br><br>return profile<br><br>Have a look at the complete list of methods and parameters available in the WhylogsDataValidator API in the SDK docs.<br><br>Call whylogs directly<br><br>You can use the whylogs library directly in your custom pipeline steps, and only leverage ZenML's capability of serializing, versioning and storing the DatasetProfileView objects in its Artifact Store. You may optionally enable whylabs logging to automatically upload the returned whylogs profile to WhyLabs, e.g.:</code> |
 * Loss: [<code>MatryoshkaLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters:
   ```json
   {
 ### Training Logs
 | Epoch   | Step  | dim_384_cosine_map@100 | dim_256_cosine_map@100 | dim_128_cosine_map@100 | dim_64_cosine_map@100 |
 |:-------:|:-----:|:----------------------:|:----------------------:|:----------------------:|:---------------------:|
+| **1.0** | **1** | **1.0**                | **1.0**                | **1.0**                | **1.0**               |
+| 2.0     | 3     | 1.0                    | 1.0                    | 1.0                    | 1.0                   |
+| 3.0     | 4     | 1.0                    | 1.0                    | 1.0                    | 1.0                   |
 * The bold row denotes the saved checkpoint.

model.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:24b8936c919b841952b05d08c369ac02d906ad7f6e4471b4141ddb8e9799e622
 size 435588776

 version https://git-lfs.github.com/spec/v1
+oid sha256:463d27a3e88a748997316367c419bbdab4d809300a5f24b9b19ba9b2dc08ffdd
 size 435588776