🗺 Prompt Management

[mikeldking]

As teams grow, it can become important to track and manage prompts independently from code for re-use inside of Phoenix and other integration points like evaluation.

Phoenix prompts will let you find, create, modify, and version control prompts for and LLMs in general.

Technical requirements

• Prompts should be versioned
• Prompts should be provider independent
• Prompts should be cashable for high availability (both server side and client side)

Spike

• [hub] PRD #5604
• [hub][spike] 4 + 1 diagram #5676
• [hub][spike] flesh out prompt management use-cases and data model
• [hub][spike] caching of prompt templates
• [hub][spike] prompts ERD #5677
• [hub][spike] prompt template model configuration tracking
• [hub][spike] phoenix.client.prompts python SDK design #5625
• [hub] scope out client-side caching #5678
• [hub] scope out server-side caching
• [hub][spike] pydantic types for messages, tools, output_schema #5719
• [hub][spike] zod types for messages, tools, output_schema
• [hub] [spike] get a good mix of prompt template examples #5737

Technical Infra

• de-couple the Phoenix client from the Arize-Phoenix package #5684

Server

• [hub] prompt template storage (e.g. database tables) #5733
• [hub] [gql] prompt listing
• [hub] mocked out history, prompt template, tags #5762
• [hub] [gql] prompt versions list #5754
• [hub] [gql] prompt template @ version #5753
• [hub] [gql] template types #5743
• [hub] [gql] prompt create #5732
• [hub] [gql] prompt template save #5801
• [hub] [gql] version tag create #5778
• [hub] [gql] version tag patch #5800
• [hub] [gql] prompt clone #5799
• [hub] [gql] ChatPromptTemplate type #5771
• [hub] [gql] prompt delete #5798
• [hub] [gql] prompt template version tag create #5763
• [hub] [gql] prompt template version tag patch #5764
• [hub] PromptTag list on a PromptVersion #5779
• [hub] [REST] /prompts/:identifier/versions/:versionId #5795
• [hub] [REST] /prompts/:id/tag/:tagIdentifier #5796
• [hub] [REST] /prompts
• [hub] [REST] rename promt tags prompt labels, rename crosswalk table prompt_prompt_labels #5788
• [hub] add auto increment to versions #5797
• [hub] prompt_version_tag field on prompt version #5846
• [hub] formalize tool definition #5848
• [hub] [gql] clone prompt mutation #5866
• [hub] [gql] filter prompts by tag (use-case)
• [hub] [gql] createPromptVersion #5880
• [hub] create and patch prompts should track author #5883
• [hub] [gql] sequence number for prompt versions #5884
• [hub] server-side validation of tool definitions via pydantic types #5887
• [hub] add types for template in create prompt mutation #5895
• [hub] graphql scalar for "identifier" #5900
• [hub] [gql] version resolver on Prompt #5909
• [hub][gql] get all version tags for a given prompt #5906
• [hub] [gql] get a version's previous version (diff) #5908
• [hub] [gql] version created at #5910
• [hub] [gql] delete prompt_version_tag #5930
• [hub] [gql] delete prompt #5931
• [hub] [gql] put the prompt version on the node interface #5939
• [hub] [gql] track the author of a tag #5947
• [hub] [gql] ability to persist a tool message #5952
• [hub] [gql] ability to persist a tool call message #5953
• [hub] add a user ID column to prompt_version_tags #5969
• [hub] [gql] name experiments run on a prompt off of the prompt name #5972
• [prompts] refactor to use third-party library to validate jsonschema fields #5987

UI

Prompt Details / Management

• [hub] [ui] prompt listing #5725
• [hub] [ui] prompt details #5741
• [hub] [ui] prompt id and version in zustand state #5735
• [hub] [ui] prompt picker / combobox in playground #5736
• [hub] [ui] first time prompt create #5734
• [hub] [ui] render a read-only view of the chat prompt template in the prompt details #5759
• [hub] [ui] prompt details aside preview of the last N versions #5761
• [hub] [ui] full prompt history list in the versions tab #5760
• [hub] [ui] display invocation parameters in the prompt details #5777
• [hub] [ui] display typescript / python code snippets for openai #5776
• [hub] [ui] prompt clone UI flow (modal) #5863
• [hub] [ui] prompt version tagging UI #5864
• [hub] [ui] render tool definitions on prompt details / prompt version details views #5865
• [hub] [ui] prompt version tag delete #5932
• [hub] [ui] prompt version tag edit (description at least)
• [hub] [ui] ability to tag a version on the prompt history
• [hub] [ui] prompt clone
• [hub] [ui] prompt tags in prompt aside
• [hub] [ui] edit prompt labels (use-cases) in prompt aside
• [hub] [ui] provider code snippet for anthropic #5951
• [hub] [ui] provider code snippet for vertex / gemini #5950
• [hub] [gql] add a sequence number / data loader #5970

Prompt Tagging / Version Management

• [hub] [ui] add pre-set tags (production, staging) so it's easy to set them #5966

Prompt Editor / Playground

• [hub] [ui] prompt save in playground #5879
• [hub] [ui] load prompt into playground state #5881
• [hub] [ui] the /prompts/:id/playground route should load a prompt into state #5882
• [hub] [ui] save prompt as new prompt in playground #5886
• [hub] [ui] load prompt invocation parameters into playground #5923
• [hub] [ui] Track + Display prompt dirty state in playground #5924

Tracing

• [hub] add semantic conventions for prompt_template.id and prompt.vendor

Analytics

• [hub] scope out prompt template analytics

Python SDK

• [hub] [python] prompts phoenix.client submodule #5997
• [hub] [python] prompt pull #5996

TypeScript SDK

• [hub] [js] phoenix-client/prompts #5998
• [js] changeset based publishing #5999
• [js] CI for phoenix-client #6000

Documentation

• [hub] [docs] prompt version tagging #5967
• [hub] [docs] prompt overview #5968
• [hub] [docs] prompt client sdk