Skip to main content

Use Prompts in Deployed Apps

Integrating versioned prompts from the MLflow Prompt Registry into your deployed GenAI applications requires careful consideration of versioning strategies, dynamic updates, and risk management. This page focuses on best practices for using prompts in a production environment, with a strong emphasis on leveraging aliases for flexibility and control.

You will learn about:

  • Configuring production applications to use registry prompts, particularly via aliases.
  • Strategies for prompt versioning in deployment (pinning vs. aliases).
  • How to update prompts referenced by aliases without redeploying your application.
  • Monitoring prompt usage in production (conceptually).
  • Utilizing aliases for A/B testing different prompt versions and for facilitating rollbacks.

Configuring Production Applications to Use Registry Prompts​

When your GenAI application is deployed to a production environment, it should load its prompts from the MLflow Prompt Registry. The recommended way to do this is by referencing a prompt alias rather than a specific version number in your application's configuration or code.

Recall that an alias (e.g., production, live-summary-prompt) is a mutable pointer to a specific, immutable prompt version. By using an alias, you can change the underlying prompt version that your deployed application uses without needing to redeploy the application itself.

Prompt Versioning Strategies for Deployment​

You have two main strategies for managing which prompt versions your deployed application uses:

  1. Pinning to Specific Versions: Your application code or configuration explicitly references a specific version number (e.g., prompts:/my-prompt/3).
    • Pros: Maximum stability and predictability. You know exactly which version is running.
    • Cons: Updating the prompt requires a code change and redeployment of your application.
  2. Using Aliases (Recommended for most production scenarios): Your application code or configuration references an alias (e.g., prompts:/my-prompt/production).
    • Pros: Flexibility. You can update the prompt version by simply changing where the alias points in the MLflow Prompt Registry, without any application code changes or redeployment. This allows for rapid iteration and hotfixes.
    • Cons: Requires careful management of aliases to ensure the correct version is live. Changes to aliases should be audited.

For most production use cases, using aliases is the preferred method due to the operational flexibility it offers.

Updating Prompts via Aliases Without Redeployment​

This is one of the most powerful features of using aliases with the Prompt Registry.

Scenario: You have a prompt invoice-parser and the alias production currently points to version 2. Your application is configured to use prompts:/invoice-parser/production.

  1. Develop and Test a New Version: You create and thoroughly test version 3 of invoice-parser, which significantly improves accuracy.
  2. Update the Alias: Using the MLflow UI or the mlflow.genai.set_prompt_alias() Python API, you change the production alias for invoice-parser to now point to version 3.
  3. Application Behavior: The next time your deployed application instances (which might have caching, see below) load the prompt prompts:/invoice-parser/production, they will automatically fetch version 3 without any code changes or restarts.

Considerations for Caching: Deployed applications might cache loaded prompts for performance. Ensure your application has a strategy to periodically refresh cached prompts or a mechanism to trigger a refresh if immediate updates are critical.

Monitoring Prompt Usage in Production​

While the Prompt Registry manages the prompts themselves, monitoring their usage and performance in production relies on your application's observability stack, ideally integrated with MLflow Tracing.

  • Log Prompt Identifiers: Ensure your application logs include the specific prompt name and version (or the alias used and the version it resolved to) for each transaction.
  • Trace Integration: If using MLflow Tracing, the mlflow.genai.load_prompt() call can automatically add metadata to your traces about the prompt URI used. This allows you to correlate application behavior, LLM outputs, and performance metrics (latency, cost) with specific prompt versions in the MLflow Tracing UI.
  • Performance Metrics: Track LLM response times, token counts, and error rates. Segment these metrics by prompt version to identify if a new prompt version has impacted performance.
  • Quality Metrics: Collect user feedback (e.g., thumbs up/down, explicit feedback) and run offline evaluations or online monitoring with LLM judges on production traffic, associating results back to the prompt versions used.

Utilizing Aliases for A/B Testing and Rollbacks​

Aliases are also invaluable for advanced deployment strategies:

  • A/B Testing: You can introduce a new prompt version to a subset of users or traffic. For example, have a separate alias like experimental-parser pointing to version 4. Your application or load balancer could route a percentage of requests to use this experimental alias, allowing you to compare its performance against the main production alias before a full rollout.
  • Easy Rollbacks: If a newly promoted prompt version (e.g., version 3 pointed to by production) is found to have issues, you can quickly roll back by simply updating the production alias to point back to the previously stable version (e.g., version 2). This can be done in seconds without a stressful redeployment.

Key Takeaways​

  • Using aliases (e.g., prompts:/my-prompt/production) in deployed applications is highly recommended for dynamic prompt updates without redeployment.
  • Updating an alias in the MLflow Prompt Registry allows for hot-swapping prompt versions in live applications.
  • Careful alias management and auditing are crucial.
  • Integrate prompt version information into your production logging and MLflow Tracing for effective monitoring.
  • Aliases facilitate A/B testing of new prompt versions and enable rapid rollbacks if issues arise.
  • Be mindful of prompt caching strategies in your deployed applications.

Prerequisites​

  • A production-ready GenAI application.
  • An established deployment pipeline.
  • MLflow Prompt Registry populated with versioned prompts and aliases.
  • (Recommended) MLflow Tracing implemented in your application for monitoring.

By strategically using prompts from the registry, especially with aliases, you can significantly improve the agility, reliability, and maintainability of your deployed GenAI applications.