Arize AI Conference: Generative AI + ML Observability

What is this event?

I was approached on LinkedIn to attend an online event on Generative AI + ML Observability, hosted by Arize AI.

The power of LLMs (Large Language Models) has taken the world by storm. Every single company is looking for ways to implement LLMs into existing and upcoming products. Even at AryaNow (the tele-psychiatry company I work for), I had the chance to implement an FAQ chatbot with GPT-3 and now GPT-4.

The purpose of this post is to summarize the key points I found interesting — as a way to bookmark these ideas for my future self and to easily share them with others.

The New Wave of ML

• Arize creates embeddings on top of OpenAI's GPT models and provides a method to score LLMs.
• Agents in the LLM space: LangChain, BabyAGI, AutoGPT.
• Arize unveiled LLM Observability (documentation) — a Euclidean-distance map of prompt-responses as a way to handle LLMs in production. The core methodology involves embeddings.
• Phoenix is their open source ML observability library (GitHub) — comparing two datasets to gauge "embeddings drift."

Five Papers Reshaping AI's Future — Brian Burns

Trend 1: Generative self-supervision

• Toolformer (AI-generated text labels): sample a massive number of API calls, filter out 90–95% of them based on their ability to explain ensuing text, then fine-tune the LLM on the enriched, annotated dataset. Scalable, dataset-agnostic, no need for human labelers.
• Related: Language Models Can Teach Themselves to Program Better and Anthropic's Constitutional AI: Harmlessness from AI Feedback.

Trend 2: LLMs as base models + encoders

• ESMFold (LLMs for protein folding): pretrain an LLM to extract protein features, incorporate the LLM base, fine-tune. Predicts 3D protein structure with accuracy comparable to AlphaFold but 60× faster.
• Related: PaLM-E: An Embodied Multimodal Language Model and Emergent autonomous scientific research capabilities of large language models.

Trend 3: Alternative LLM architectures

The bottleneck is transformer context: a single pass of an attention layer requires n² compute (n = context length), so information-dense contexts — video, genetics, long-form text — can't be processed. Approaches to watch:

• Hungry Hungry Hippos (Stanford NLP): state-space model architecture for LLMs using Fast Fourier Transforms to compute long convolutions
• Hyena Hierarchy (Stanford NLP, April 2023)
• RWKV (open source research community)
• Recurrent Memory Transformer

The Future of the ML Lifecycle (Panel)

With Manu Sharma (Labelbox), Robert Nishihara (Anyscale), Mike Del Balso (Tecton), Chris Van Pelt (Weights & Biases), and Aparna Dhinakaran (Arize). The main takeaway: a lot of LLM work at major companies — fraud models, recommendation systems — is still in a prototyping phase.

Embeddings at Spotify's Scale — Ian Anderson

Spotify operates at 500M+ monthly active users across 180+ markets, with 100M+ music tracks and 5M+ podcasts, blending editorial and algorithmic approaches.

The 2015 approach: word2vec on playlists to learn music embeddings, Annoy for approximate lookup, user embeddings from prior listening, filtered for previous listens.

Today embeddings power Discover Weekly, Release Radar, artist radio, search, Niche Mixes, and DJ AI. The interesting challenges:

• Recency (music is not language): cold-start and long-tail problems with a bigger, faster-growing vocabulary — and outsized-importance items ("stopwords vs. Taylor Swift"). Solutions: static versioning, content-aware embeddings, warm starts.
• Exogeneity: cultural events resurrect old songs ("Running Up That Hill," "Dreams") in ways no model predicts. Solutions: ample evaluation and monitoring, tightly bounded schemas, batch-aware systems.
• Delivery at O(1B) MAU scale: extremely tight latency budgets and trillions of logs per day. Solutions: leaning into embeddings, probabilistic data structures, approximate retrieval, event-driven architecture.
• Multi-domain: Spotify is more than music — text-to-speech, speech-to-text, prosody. Solutions: standardized evaluation, foundation models, decomposability.

A Practical Perspective on Using LLMs — Rajiv Shah (HuggingFace)

The framing question every executive is asking: what is your ChatGPT strategy? In the near term: nothing major will change.

Key concepts: zero-shot and few-shot (in-context) learning, RLHF, and parameter-efficient fine-tuning (PEFT).

The easy decisions — for high-value use cases, a data science team will still build a dedicated model:

• Domain-specific tasks → dedicated model
• High scale / low latency → dedicated model
• Explainability / model risk → dedicated model

Factors when choosing LLM vs. dedicated model: predictive performance, scaling, inference speed, data privacy, explainability, model risk, cost, development and retraining time, and operationalization in your enterprise.

Three main questions for businesses: Can you train your own LLM? Should you go open source? Should you leverage commercial LLMs via API? BloombergGPT (50B parameters, trained on 345B general-purpose tokens plus 363B tokens of Bloomberg's own data) shows building your own is replicable — if you have the time, money, and a proprietary dataset.

Bringing LLMs to Enterprise Data — Jerry Liu (LlamaIndex)

LlamaIndex has three components:

1. Data connectors (LlamaHub): ingest any kind of data from anywhere — APIs, PDFs, docs, SQL — powered by a community-driven hub with growing multimodal support.
2. Data indices: structure your data for different use cases, abstracting away common boilerplate.
3. Query interface: feed in a prompt, get a knowledge-augmented output — semantic search, summarization, text-to-SQL, and synthesis across heterogeneous data sources.

Cracking the ML Interview — Dat Ngo & Claire Longo

• Hiring managers look for enthusiasm — genuine passion for a specific use case or model type — and a mentorship match between manager and data scientist.
• Core skills: technical (Python, data manipulation, visualization, model understanding), problem-solving (working autonomously with ambiguity), and communication (explaining why your work matters to non-technical stakeholders).
• Portfolio advice: a GitHub the hiring manager can actually read, and projects you can discuss in depth — not just at a high level.
• Resume advice: statements that show business ROI, and leadership signals.
• The market is hard right now; the network is the most valuable resource. High-frequency applying can mean 200 applications per callback.

Intro to Relative Representations — Luca Moschella

Training the same autoencoder twice on MNIST produces different latent spaces — but relative representations make them comparable. The algorithm: select a subset of the training set as anchors, encode a sample, and represent it by its similarity to each anchor. The representation is computed independently for each space, its size depends on the number of anchors, and the anchor/similarity choices determine its properties. This exposes semantic invariances across models — e.g., FastText vs. Word2Vec embeddings of the same words end up looking very similar in relative space.

The Most Desired Skills for MLEs in 2023 — Chip Huyen, Han-chung Lee, Gourab De

• "A data scientist who codes" → machine learning engineer; MLE culture is more engineering-driven.
• ML and DS are splitting into niches, and companies hire for previous experience in the specific modality (tabular — churn, fraud — vs. unstructured — generative AI, segmentation).
• End-to-end project experience is the must-have: from the germination of an idea all the way to delivery. Deploy something to an API people can actually call. Toy projects (Titanic, stock prediction) aren't a good signal for industry.
• Business sense matters: problem identification and product sense — can you speak from the user's perspective?

A Framework for Data Storytelling — Wendy Foster

Building from Drew Conway's data science Venn diagram (creative problem-solving + technical + domain skills), a data story has three parts: the data question (what are we trying to answer?), the finding (description + visualization), and the recommendation (next steps based on the data).

Story component	Business component	Requirements
Main character	Business problem	Technical and domain
Setting	Problem space / context	Hypotheses and results
Narrator	Accessible language	Clear, non-jargon
Plot	Action-based recommendations	Supported decision making

An Autonomous Agent is Born — Yohei Nakajima (BabyAGI)

• BabyAGI started with Yohei exploring HustleGPT while working at his VC firm.
• Interesting uses so far: web-based implementations, "god mode," and BabyAGI powering NPCs in games.
• The prioritization agent began as a de-duping agent, and is evolving toward hierarchical and parallel task handling.
• His take on where agents are going: agents aren't new — first internal tools will be built from agents, then external tools for simple tasks, and eventually general-ability agents on an unknowable timeline.

Future of Agents — Harrison Chase (LangChain)

LangChain is a framework for developing applications powered by language models that are data-aware (connected to other sources of data) and agentic (able to interact with their environment).

• Unless agents are narrow, they are just cool Twitter demos — narrowly focused agents seem viable.
• Agents allow handling more edge cases (e.g., adding metadata filters at query time for better retrieval).
• Where agents are going: more generic tool usage, long-term memory (the "generative agents" Westworld paper), and LLM-assisted evaluation — putting LLMs in the loop where human feedback used to be the only metric.
• On observability for agents: just looking at inputs and outputs isn't satisfying; in the short term, people have to manually inspect them.