OpenAI has introduced several new features and models for developers and startups using their API. These include the launch of the 01 model out of preview, which has been used for building agentic applications, customer support, and financial analysis. Key features launched include function calling, structured outputs, developer messages, and reasoning effort parameters. Function calling allows models to interact with backend APIs, while structured outputs ensure models adhere to specific formats. The reasoning effort parameter optimizes the model's thinking time, saving resources on simpler tasks. Additionally, vision inputs have been introduced to aid in fields like manufacturing and science. OpenAI also announced the introduction of preference fine-tuning, a method that optimizes models based on user preferences, enhancing performance in areas like customer support and content moderation. This method uses direct preference optimization, allowing developers to guide models towards preferred behaviors. The company has also improved the real-time API with WebRTC support, reducing latency and simplifying integration. New SDKs for Go and Java have been released, and the cost of GPT-4 audio tokens has been reduced. These updates aim to enhance the developer experience and expand the capabilities of applications built on OpenAI's platform.

OpenAI's structured outputs feature, launched in August, ensures that AI-generated outputs strictly adhere to JSON schemas provided by developers. This advancement addresses previous issues with unreliable outputs from large language models (LLMs), which often included unnecessary text or incorrect data types. Structured outputs are available in two modes: function calling and response format parameter. Function calling allows developers to define tools using JSON schema, ensuring the model outputs valid JSON. The response format parameter is useful when the model responds directly to users, maintaining the specified format. The feature eliminates errors by constraining outputs to match the provided schema, enhancing reliability in applications like AI-powered recruiting tools and AI glasses. The engineering behind structured outputs involves constrained decoding, token masking, and supporting a wide subset of JSON schema, including recursive schemas. This ensures fast inference and reliable outputs. OpenAI's research improved model accuracy in following complex schemas, achieving near-perfect results when combined with constrained decoding. The API design prioritizes explicit constraints, requiring developers to specify additional and required properties, and maintains property order to improve output quality. Since its launch, structured outputs have significantly improved application reliability, reducing errors and hallucinations, as seen in companies like Shopify.

Genmab, a biotech company, is leveraging AI to streamline the clinical trial process, which is traditionally lengthy and costly. The company has developed a framework called CELI to automate the generation of regulatory documents required for clinical trials. These documents, which detail patient information and trial data, are typically labor-intensive and require high accuracy. CELI uses a language model to process natural language inputs, plan tasks, and self-correct, ensuring 100% accuracy in document generation. This system significantly reduces the time needed to compile documents, transforming a process that could take hours into one that takes minutes. By accelerating documentation, Genmab aims to shorten clinical trials, allowing faster access to treatments for patients with serious diseases.

Caitlyn Augustine from DataKind highlights the critical need for high-quality data in humanitarian efforts, noting that 300 million people require assistance globally, with a $46 billion funding gap. DataKind collaborates with humanitarian organizations to identify data access challenges and explores solutions like generative AI for metadata prediction. Despite the existence of metadata standards like hexel, adoption is low, leading to interoperability issues. Generative AI can improve metadata tagging accuracy, with recent tests achieving over 95% accuracy for common metadata like locations and dates. The project aims to make data processing cost-effective and efficient, allowing humanitarian organizations to handle large data volumes with minimal resources. The initiative is part of a broader effort to create a comprehensive humanitarian data system, including an AI assistant for rapid data access and response.

The speaker, Tilde, discusses the biases inherent in large language models due to their training on human data. They highlight research from Anthropic and Princeton University that explores how these models can exhibit biases, such as positive discrimination towards women and non-white individuals, and negative age discrimination. The Anthropic study used prompts to test bias in decision-making, finding that reminding models that discrimination is illegal and instructing them to ignore demographic information reduced bias. The Princeton study adapted implicit bias tests for models, showing that explicit decision-making prompts reduced bias. Practical applications include improving prompt engineering by adding contextual information and instructing models to ignore demographic data, as demonstrated in writing unbiased reference letters. Tilde emphasizes not using these models for high-stakes decisions and suggests using tools like LaunchDarkly for testing prompts and models.

The real-time API, recently launched in public beta, allows developers to create applications with low-latency voice interactions using a single API. This API unifies capabilities such as speech recognition, transcription, and text-to-speech, which previously required stitching together multiple models. The API natively understands and generates speech, eliminating the need for converting modalities into text. This advancement enables smoother, more natural conversational experiences, as demonstrated through various examples like voice-driven web browsing and interactive educational apps. The API supports real-time streaming of audio and text, allowing for immediate responses and natural interruptions. Developers can integrate this API into their applications to create immersive, voice-interactive experiences. The video also highlights the API's ability to handle tool calls, enabling dynamic interactions and data integration within apps. Additionally, the API's cost efficiency is improved through prompt caching, reducing costs for cached inputs significantly.

Altera.AL, led by Robert Yang, is focused on creating digital humans that can live, love, and grow alongside humans. The company aims to build agents with fundamental human qualities such as emotion, coherence, and possibly consciousness. These agents are designed to collaborate and progress with humans over long time horizons, potentially transforming productivity to levels comparable to entire countries. Altera.AL's Project Sid explores the potential of autonomous agents by simulating environments like a Minecraft server, where agents develop emergent economies, religions, and social structures. The company addresses challenges in long-term agent progression, such as data degradation and looping, by employing concurrent, context-dependent modules inspired by brain architecture. This approach allows agents to process information at different timescales and make coherent decisions, enhancing their adaptability and efficiency. Altera.AL's research aims to achieve a future where multi-agent collaboration is seamless and impactful.

Garrick Toubassi, co-founder of Mindtrip, introduces the platform as an AI-powered travel solution designed to assist users throughout the entire travel lifecycle, from inspiration to booking. Mindtrip addresses the challenge of converting static travel content, like blog posts and images, into dynamic, actionable travel plans. By integrating multimodal inputs, such as text, images, and videos, Mindtrip enhances the travel planning experience. For instance, users can input a blog post or an image, and Mindtrip will generate a structured itinerary, complete with maps and points of interest. The platform leverages the Chat Complete API to process text and images, and employs tools like FFmpeg and OpenAI's Whisper model for handling video content. This approach allows Mindtrip to transform various content types into useful travel plans, bridging the gap between inspiration and action. Toubassi also mentions experimenting with the new Realtime API, which offers potential for real-time audio integration, further enhancing the platform's capabilities.

The speaker, originally from Singapore but residing in the U.S. for 15 years, aims to transform Singapore into an AI engineering nation. He highlights the power of foundation models available today and the importance of understanding AI engineering. The talk covers the rise of the AI engineer, a specialist role for software engineers building with AI, and the speaker's podcast focusing on AI engineering. He shares insights on building AI agents, emphasizing the components of agents as LLM plus memory, planning, and use. The speaker discusses the importance of having a clear map of jobs to be done for building LLM infrastructure, recommending tools like Gateway, Ops tool, and rag framework. He highlights the significance of memory and knowledge in AI, mentioning tools like Vector DB and knowledge graphs. The talk also covers planning and multi-agent systems, recommending resources like OpenAI's papers and projects. The speaker demonstrates a demo of building a Space Invaders game using AI agents, showcasing the potential of human-AI collaboration. He encourages leveraging the agent stack to build innovative solutions and emphasizes the potential in Singapore for AI development.

The discussion focuses on how Genmab, a biotech company, uses AI agents to streamline the clinical trial process, which is traditionally long and costly. The AI framework, named CELI, is designed to automate the generation of regulatory documents required for clinical trials. These documents, which detail patient information and trial data, are typically labor-intensive and require high accuracy. CELI uses a language model to process natural language inputs, plan tasks, and self-correct, ensuring 100% accuracy in document generation. This automation significantly reduces the time required to compile these documents, potentially shortening clinical trials by months, which can expedite patient access to new treatments. The system is capable of handling complex tasks by breaking them down into smaller, manageable sections, ensuring precision and efficiency. The ultimate goal is to improve the speed and accessibility of clinical trials, benefiting patients with serious diseases by providing quicker access to new drugs.

Jerry and Daniel from SAA present their AI platform designed to solve access to knowledge by integrating structured and unstructured data. The platform supports complex workflows, allowing users to interact with data from multiple sources without leaving the interface. They demonstrate how the platform can handle tasks like listing CRM opportunities and updating Salesforce records through a unified chat interface. Daniel explains the importance of tool sequencing and user feedback in ensuring successful agent interactions. They found that providing instructions in user messages significantly improves workflow success, especially for complex tasks. The platform uses a tool set and a router LLM to manage workflows, ensuring users have the best tools for their needs. High integrity tool responses and user feedback are emphasized to enhance collaboration and productivity.

Video GPT is an AI-powered application designed to simplify the video creation process, available on the GPT store. The platform allows users to create professional-quality videos quickly by leveraging large language models to generate scripts and customize visuals. The speaker, Saba, CEO and co-founder of Ved, shares her journey from art school to developing this innovative tool. She highlights the challenges of traditional video creation and how Video GPT addresses these by streamlining the process from idea to finished product in under a minute. The tool has gained popularity, becoming the top video application on the GPT store, and significantly contributing to Ved's customer base. The platform's success is attributed to its user-friendly interface and the ability to create videos without needing extensive editing skills. Saba also introduces a new feature, "Slide to Video," which converts documents like slide decks into videos, further enhancing the platform's versatility.

Grab, a leading super app in Southeast Asia, is enhancing its mapping services through community-based mapping and AI technologies. Initially, Grab Maps was developed due to the inadequacy of third-party apps in providing localized data. Grab Maps now serves both internal needs and external businesses across Southeast Asia. The approach involves collecting street-level imagery using 360° cameras from its driver network, extracting details like traffic signs and road accessibility to build detailed maps. Recently, Grab has adopted OpenAI's vision fine-tuning capabilities to improve data matching for traffic signs, addressing challenges like intricate geometries and visual occlusions. This involves using a small fine-tuning dataset combining street-level imagery and map tiles to accurately match traffic signs to roads, enhancing the reliability of their maps.

Mike from Paloa discusses the transformation of contact centers using AI, specifically OpenAI's GPT-4, to enhance customer service. The goal is to make interactions as natural as talking to a friend, using AI agents to complement human agents rather than replace them. This involves creating personal AI agents that handle unique conversations and resolve issues effectively. The AI agent lifecycle is crucial, focusing on safe and responsible deployment, including design, integration, testing, and scaling. Paloa has launched an AI agent management platform to support this process, emphasizing simulation and evaluation to ensure reliability and compliance. The platform allows for the configuration of AI agents to handle various customer personas and scenarios, ensuring effective and compliant interactions. The future vision includes AI-first contact centers where AI agents handle most tasks, with human agents acting as supervisors and coaches, ensuring a seamless transition and improved customer experience.

Amperity, a customer data cloud company, has developed AmpAI, a tool that leverages OpenAI's models to help brands make sense of complex customer data. AmpAI is designed to translate natural language queries into SQL, enabling non-technical users to interact with and visualize their data. The tool addresses challenges such as siloed data and the need for SQL knowledge by providing a user-friendly interface. AmpAI is particularly useful for marketers like Lauren from Acme Retail, who need to identify high-value customers but face difficulties due to disparate data systems and complex database structures. The tool employs a multi-step process to manage context, including ranking database tables and identifying key fields, to generate accurate SQL queries. This approach allows AmpAI to work across various industries and brands, adapting to different data schemas and requirements. The implementation of AmpAI has led to a significant increase in data queries run by Amperity's customers, demonstrating its impact and effectiveness in improving data accessibility and analysis.

The o1 model is a reasoning model trained with reinforcement learning to refine its thinking strategies and correct mistakes. It is particularly effective in solving difficult problems by iteratively improving its strategies. The model represents a new paradigm in AI, offering enhanced reasoning capabilities that can significantly outperform previous models like GPT-4o in specific domains such as math and coding. The video emphasizes the importance of considering what becomes possible with improved reasoning and how this can influence future developments. Practical applications of o1 include solving complex math and code problems, medical accuracy detection, and serving as a brainstorming partner in various fields. The o1-preview and o1-mini models are highlighted for their performance in specific tasks, with o1-mini being optimized for speed and performance in math and coding tasks.

The discussion highlights OpenAI's progress towards AGI, emphasizing a gradual, non-binary approach to achieving it. They introduce a levels framework to categorize AI capabilities, with current models reaching Level 2 (reasoners) and aiming for Level 3 (agents). The conversation stresses the importance of research in driving product development, with OpenAI committed to iterative deployment to ensure safety and alignment. They acknowledge the challenges of balancing innovation with safety, especially as AI models become more capable. The potential of AI agents is explored, with expectations for significant advancements by 2025. OpenAI also addresses concerns about alignment and safety, emphasizing their commitment to building safe systems and iteratively deploying models to learn and adapt. They discuss the role of AI in government and open-source contributions, highlighting partnerships and the potential for AI to improve efficiency and solve global issues. The conversation concludes with a vision for future AI interactions, emphasizing the transformative potential of AI in everyday life.

The presentation by Nina, a research engineer at Torus, highlights the use of LLMs in clinical settings to alleviate clinician burnout by reducing time spent on computer tasks. Torus, an LLM-powered application, allows clinicians to focus more on patient care by automating documentation processes. The video explains the development of a platform that breaks down complex workflows into smaller, manageable blocks, enabling clinicians to design and evaluate workflows themselves. This approach not only speeds up the process but also ensures clinical safety by minimizing errors such as hallucinations and omissions. The platform's iterative framework allows for continuous improvement and safe deployment of new models and workflows. Additionally, the creation of a large-scale dataset of LLM-generated clinical documentation errors aims to automate error detection and enhance product safety.

Karthik Narasimhan from Sierra introduces TAU-bench, a benchmark designed to evaluate AI agents in real-world scenarios. The benchmark addresses the challenge of assessing AI agents' performance by simulating dynamic, realistic conversations using large language models (LLMs). TAU-bench combines elements from dialog systems and agent benchmarks to create a comprehensive evaluation tool. It uses LLMs to simulate user interactions, allowing for scalable, cost-effective, and repeatable testing. This approach helps measure the reliability of AI agents by running the same scenarios multiple times, which is difficult with human testers. The benchmark introduces a new metric, pass^k, to assess an agent's performance across repeated scenarios. Initial results show significant room for improvement in AI agents' reliability, highlighting the potential of LLM-based simulators in enhancing AI evaluation processes.

Mark, OpenAI's head of research, discusses the evolution of AI, emphasizing the technical depth in Singapore and the importance of reasoning in AI models. He highlights the advancements in image and speech generation, noting their impact on AI's capabilities. Mark addresses the challenges of achieving AGI, explaining that while economic value is evident, defining AGI varies. He stresses the importance of utility and reasoning in AI development, suggesting that reasoning enhances safety by allowing models to reflect before responding. Mark also discusses synthetic data's role in improving model training, particularly in image generation, and addresses concerns about AI hitting a 'wall' in development. He reassures that OpenAI is committed to research and safety, focusing on exploratory projects with high conviction. The conversation also touches on the potential of AI to transform industries and the importance of interdisciplinary collaboration in shaping AI policy and safety. Mark concludes by emphasizing the supportive and innovative culture at OpenAI, which fosters impactful research and development.

The speaker, Ally, co-founder and CEO of Cosign, introduces Genie, an AI engineer designed to autonomously handle software engineering tasks. Genie was developed by fine-tuning GPT-40 with synthetically augmented real-world data to mimic how developers complete tasks. The speaker highlights the importance of fine-tuning, which, although underutilized, is crucial for achieving high-quality performance in AI models. Fine-tuning allows models to specialize in specific tasks with relatively few examples, but requires detailed data handling and cleaning. The speaker also discusses the use of custom reasoning traces to enhance the model's ability to think like a human, particularly in complex tasks like software engineering. This involves using models like 01 to generate reasoning traces that help the AI model understand the context and reasoning behind human decisions. Additionally, the concept of self-play is introduced as a method to generate training examples that are difficult to obtain in the real world. Self-play involves a player model and a supervisor model working in a loop to simulate decision-making processes. The speaker emphasizes the importance of data quality and the potential of fine-tuning to improve model performance significantly.

Jared, leading the AI team at Vercel, introduces v0, a generative UI AI tool that bridges design and coding. The tool aims to democratize software creation, allowing anyone to build personalized applications without extensive coding knowledge. This shift is termed the 'era of personal software,' where sophisticated code generation and AI tools enable individuals to create tailored solutions. Jared demonstrates v0's capabilities through a text-to-speech app and highlights its potential to replace complex tasks traditionally handled by business analysts with automated app generation. He emphasizes the tool's ability to foster creativity and collaboration within organizations, transforming the role of developers and making software development more accessible. The presentation concludes with a vision of democratized creativity, where everyone can contribute to software creation, akin to the idea that 'everybody can cook.'

The conversation highlights the limitations of open-source tools like OpenAPI Generator, which lack essential features such as streaming, crucial for certain applications. The speaker explains that their company provides added value by integrating these features into their SDKs, allowing for custom code modifications and structured outputs. This customization enables users to make arbitrary changes to the SDKs, similar to managing a regular repository. The discussion also touches on the importance of creating appropriate abstractions in SDKs. While thin wrappers over HTTP APIs ensure full support and easy mapping from API documentation to SDK usage, certain abstractions like pagination and auto-retries are beneficial. However, essential details like headers and response times should remain accessible to users. The conversation concludes with a caution against automatically generating code that may not meet quality standards, particularly in languages like Python, where code should adhere to idiomatic practices.

Anna Dixon, an applied research scientist at Dimagi, explains their project funded by the Bill and Melinda Gates Foundation, which aims to use large language models (LLMs) for family planning education in Kenya and Senegal. The project focuses on fine-tuning GPT-4o mini and GPT-4o for health education chatbots in low-resource languages like Sheng, a Swahili-English slang. Initial attempts using zero-shot and few-shot prompting were ineffective, leading to the injection of Sheng sentences as a style guide, which improved language quality but was costly and slow. The team then implemented a machine translation layer to translate English responses into target languages, enhancing modularity and evaluation capabilities. Fine-tuning improved translation quality significantly, with GPT-4o mini's spBLEU score increasing from 22.21 to 65.23. The project is expanding to other languages, showing promising preliminary results in Chihchewa, with fine-tuned models doubling initial scores. The approach demonstrates cost-effective improvements in language model performance for health education in low-resource settings.

The speakers, Colin Jarvis and Jeff Harris from OpenAI, discuss the challenges of scaling AI applications as user bases grow. They emphasize the importance of optimizing for accuracy first, using the most intelligent models until accuracy targets are met. Once accuracy is achieved, the focus shifts to optimizing latency and cost. They provide practical techniques such as prompt engineering, retrieval-augmented generation (RAG), and fine-tuning to improve accuracy. For latency, they suggest breaking down total request latency into network latency, time to first token, and time between tokens, and offer strategies to optimize each. Cost-saving measures include using prompt caching and the BatchAPI, which offers significant discounts for asynchronous processing. The speakers highlight the importance of balancing these factors to build efficient and effective AI applications.

Caitlin Augustin from DataKind discusses the critical need for timely and high-quality data in the humanitarian sector, where 300 million people require assistance and funding gaps reach $46 billion. DataKind aims to address these challenges by leveraging generative AI to improve metadata prediction in humanitarian datasets, which often lack interoperability due to inconsistent or missing metadata. Despite the existence of the HXL metadata standard, adoption has been low due to the time-consuming and error-prone nature of manual labeling. DataKind's approach involves using AI models like GPT to automate metadata tagging, achieving over 95% accuracy for common metadata such as locations and dates. This automation not only reduces manual effort but also enhances data quality, enabling more effective humanitarian responses. The project has been successful in meeting accuracy, cost, and time targets, unlocking thousands of variables for humanitarian use. The initiative is part of a broader effort to create a Humanitarian AI Assistant that integrates harmonized data for rapid response, co-created with humanitarian stakeholders.

OpenAI's DevDay introduced several advancements aimed at empowering developers with cutting-edge AI tools. The event showcased the new o1 model series, which excels in reasoning and problem-solving, offering both a preview and a mini version for different use cases. Developers like Cognition and Casetext have already tested these models, demonstrating their potential in coding and legal applications. Additionally, OpenAI launched the Realtime API, enabling low-latency speech-to-speech interactions, which can be integrated into applications for enhanced user experiences. The event also highlighted vision fine-tuning capabilities, allowing developers to improve image-based tasks. OpenAI emphasized its commitment to reducing AI deployment costs, introducing prompt caching for cost efficiency and model distillation tools to create smaller, more efficient models. These innovations aim to make AI more accessible and customizable for various industries.

Sam Altman, CEO of OpenAI, emphasizes the importance of reasoning models in advancing AI capabilities. He believes these models will significantly contribute to scientific advancements and complex coding tasks. Altman discusses OpenAI's strategy to improve models continuously, suggesting that businesses should align with this trajectory rather than patching current model shortcomings. He also highlights the potential for AI to create trillions of dollars in new market value by enabling previously impossible products and services. Altman touches on the development of no-code tools, suggesting that while initial efforts will enhance productivity for those who can code, high-quality no-code solutions will eventually emerge. He also addresses the role of open-source models, suggesting a balanced ecosystem where both open-source and integrated services coexist. Altman reflects on the challenges of rapid growth and the importance of focusing on long-term strategic goals rather than short-term gains. He acknowledges the need for diverse talent and the potential of AI to unlock human potential globally. Altman also discusses the complexities of AI development, including the need for specific models for agentic tasks and the evolving nature of AI systems.

The speaker, Alden, a Solutions Engineer at Dust, introduces a unified text-to-SQL solution that allows users to query data from different sources such as data warehouses, spreadsheets, and CSVs using AI-driven assistants. Dust, an AI operating system, enables the creation of specialized assistants that can access company data and integrate with platforms like Zendesk. The solution allows users to perform complex SQL queries without needing SQL knowledge, by using natural language to request data visualizations and analyses. Alden demonstrates how an assistant can query a Snowflake data warehouse to visualize data, such as the average number of messages sent on the Dust platform, and create interactive graphs. The system can also merge data from different sources, like Google Drive and CSV files, to provide insights into user roles and activity. The architecture involves converting various data inputs into a unified CSV format, which is then processed by a language model to generate SQL queries. This approach simplifies data analysis for non-technical users, allowing them to perform business intelligence tasks efficiently.