GenAI, understood as a class of models capable of generating human-like, high dimensional outputs like text, image or sound, is experiencing great success and explosive growth [1, 2, 3]. However, this has also quietly given rise to a critical problem that permeates applied GenAI in its entirety – what we call Evaluation Derangement Syndrome (EDS). In short, EDS is the problem of the widespread lack of a rational approach to and methodology for the objective, automated and quantitative evaluation of performance in terms of generative model finetuning and prompt engineering for specific downstream GenAI tasks related to practical business applications.

In this post, we analyze EDS from a practical, applied, business perspective, drawing from our rich experience in GenAI development for business, both in image generation (with diffusion models and GANs) and LLMs in code generation, retrieval systems, voice assistants, etc. We’ll explore the underlying causes (both technical and business), and examine the consequences it may have for the GenAI community and beyond.We analyze its intricate relationship with GPU inequality [4] and address how the ‘GPU-rich’ (a handful of firms with thousands of the strongest GPUs, as well as resources like data, engineers, and labelers) approach the problem of GenAI evaluation, in contrast to the harsh realities of the ‘GPU-poor’ (everyone else, really). We also discuss the fundamental insufficiency of the ‘pseudo-evaluation’ approaches used by the GPU-poor and sketch a potentially more rational path forward for them: Evaluation-Driven Development (EDD). The subsequent post will take a deep dive into the nitty-gritty practicalities of this approach, drawing from our extensive experiences with Diffusion Models and LLMs in the ‘GPU-poor’ landscape. Enjoy!

GenAI evaluation in the realm of the GPU-poor

For fundamental technical reasons, GenAI does not naturally lend itself to any obvious and reliable analogues of quality monitoring tools (like F1 score, accuracy, precision, etc.) that all data scientists live and breathe when practicing traditional ML. Then there are business pressures to deliver at an extreme tempo, typical of the heated ‘hype economy’ driven by the fear of taking over the target niche in the AI revolution, contributing to the ‘produce fast, test later (i.e., never)’ approach.

Additionally, almost all GenAI contain several evaluation dimensions lying on a spectrum from soft (subjective) to hard (objective). To give specific examples from our own GenAI projects and practice, the evaluation of:

• a chat/assistant includes subdimensions like helpfulness, friendliness, or even political correctness (subjective), vs factual correctness that can be measured in a ‘hard’ quantitative manner for some of the questions/answers (objective);
• a retrieval system may include retrieval coverage correctness (‘hard’ in many cases) and conversation/summary style (soft), similar to the assistant;
• code generation can be broken down into (hard) code correctness/test passing and (soft) code clarity;
• diffusion-based face generation may consist of (soft) image attractiveness, similarity to the desired target, but also (hard) domain adherence, i.e., % of generated images actually containing a face.

This mix of soft and hard evaluation criteria poses technical difficulties (analyzed in this section), and the resulting simple truth is this: almost all GPU-poor researchers working on GenAI applications today work without any rational, objective, quantitative, repetitive framework to evaluate their work or to inform choices – their own or that of the business decision-maker – that depend on them. When dealing with our day-to-day dilemmas we all depend on subjective, arbitrary gut feelings built in short, selective inspections of the systems we train. These are (at best) accompanied by weak numeric pseudo-evaluation (‘broken evaluation methods that put more emphasis on style rather than accuracy or usefulness’ [5]), that do almost nothing to evaluate our specific business capacities (I’m looking at you, leaderboard rankings [6], BertScores [7], BLEUs [8], etc.). The former are used more as a rationalization than an actual trustworthy indicator of performance and a primary driver of our research.

This is truly astonishing when considering the rigorous objective evaluation practices ingrained in traditional ML (Image 1). More shockingly, everybody in the field seems to accept this and move along: if we move fast (as in the production of more and more GenAI), we are fine with not checking the direction (as in objective QA, and in comparison with competition or alternative approaches).

This is not an academic or theoretical issue, but a weighty real-life problem with business, technical, and human consequences. Again, deepsense.ai’s wide experience allows us to share a specific story that many GenAI researchers can relate to. One of our valued customers asked us to develop a code-generating solution for a somewhat niche language (think GitHub Copilot’s [9] competition for this language). Even though the team we established consisted of elite LLM experts, the task proved very challenging. Due to the limited time allocated for creating the evaluation pipeline, it relied solely on the BLEU-based evaluations. The main effort went directly into generative model creation itself. This, combined with the release of the new, possibly superior base models over the duration of the project, led to serious internal evaluation issues.

To cut a long story short – our team was highly competent and worked very hard, but had no reliable way of determining if any improvement had taken place. Considering our BLEU-based evaluations, we thought this may not be the case! Luckily, the client’s own internal evaluation at the end of the project was very positive. Apparently, according to the client, our model was visibly superior to both the foundational models and GitHub Copilot. Good enough for us, and another job well done! But, to be fair, we have no idea how objective and extensive this ‘client-based evaluation’ was, or whether this will work next time. Being professionals, we prefer to make our own luck instead of being part of EDS. Today’s GenAI development is full of similar stories, rarely with a happy ending like ours.

In summary, EDS is a serious, practical issue affecting all areas of GenAI, most notably LLMs [10, 11], and image generation (GANs [12], Diffusion Models [13]). Its reach will only grow in the future, together with GenAI use cases. Generative AI can create jokes [14], stories [15], poems [16] and beautiful images [17] for a continuously growing number of applications, but our ability to evaluate GenAI is constantly falling even further behind. Given the scale of the pandemic and the technological, economic, and political impacts of GenAI, EDS truly is a critical issue. Let’s now investigate the underlying causes of this situation.

EDS – business causes

EDS emerges in the GPU-poor domain due to a complex interplay of factors—some technical and others soft, encompassing the economic, psychological, business-strategic, and organizational dimensions. Before delving into the technical reasons behind this situation, let’s take a look at the broader realities of the GenAI business-economic ecosystem contributing to EDS amongst the GPU-poor.

The first EDS-inducing condition is that, since at least 2022, the GenAI business ecosystem has operated permanently in ‘hyped-economy’ mode [18, 19]. In the GPU-poor realm, this means the red-hot fervor of startups [20] competing in a race to quickly find their niche in the GenAI revolution. Or at least respond to the marketing pressure to be a part of GenAI… A pervasive sense of ‘it’s now or never’ exerts enormous pressure on businesses. CEOs of even modest startups aspire to innovate within GenAI, sometimes despite limited technical understanding and unreliable intuitions concerning the likely future GenAI developments and their risks to early adopters [21]. The unrelenting drive to produce GenAI en masse is fueled by sky-high VC investments [22, 23] partly streaming from a ‘pay-to-participate’ strategy to keep a horse in the big GenAI race. A strategy that is at times questionable, considering that the chance to join the GPU-rich is likely far-fetched at this point [24].

The second EDS-relevant characteristic of the GenAI ecosystem is the astonishing tempo of the general-purpose model/innovation of releases by the GPU-rich (proprietary or open-source). This creates significant potential to undermine or eliminate early GenAI adopters – their business strategies are potentially threatened each time we are exposed to a release that redefines the GenAI landscape and boundaries of what is possible. This disruptive tendency manifests every few months and shows no sign of slowing down, with the recent releases of Llama 2 [25] and Mistral [26] (the great hopes of open source NLP [27, 28]) and two proprietary game-changers seemingly just around the corner: Gemini [29] and GPT-5 [30].  As a result, the GPU-poor research and do business on ‘moving sands’. As a side-effect of any release by the GPU-rich, their research projects may likely be outdated before completion. Indeed, rapid and unpredictable innovations of the GPU-rich will repetitively wipe out niches targeted by countless GPU-poor early adapters, before any real chance of a return on investment.

It is not merely that “it’s totally hopeless to compete with us on training foundation models you shouldn’t try, and it’s your job to, like, try anyway” [31], as Sam Altman accurately summed up the situation of the GPU-poor vs GPU-rich. It’s also safe to assume that once the battle for supremacy for the ‘foundation models’ is more advanced, the same fate will befall utilities and applications of these models, where many of the GPU-poor hope to find a place for themselves. Think retrieval systems, agents, specialized finetunes, and other services built on and around the foundational models. Whenever serious consequences (in USD) are involved, the GPU-rich will (in time) provide these ‘auxiliary’ services out of the box.

Hence, the GPU-poor’s fear of this ‘side effect’ of business eradication is contributing to EDS by creating additional pressure to ship fast and skip evaluation, which is not perceived as a critical business goal.

The third ‘soft’ cause is that fighting EDS among the GPU-poor may not be a primary interest of any key players involved. The main contributors to the GenAI revolution (the GPU-rich) have a proper way of evaluating GenAI (see HFRL section), but no incentives to develop or share alternatives for those who cannot afford it. In fact, from the perspective of GenAI’s future, they may see the current practices of the GPU-poor as inconsequential and irrelevant, and can leave them to their dubious practices. Sure, GPU-poor startups built around GPU-rich models may turn out to be successful from an economic standpoint (and deepsense.ai is here to make sure of it!), but they are not likely to shape the GenAI landscape in any major way over the course of a decade – this role belongs to the GPU-rich. Other parties are also fine with EDS; ML training providers are happy to train any model, regardless of its performance, as some developers and consultants are to create them regardless of the actual capacity to validate quality. And so the EDS machine may continue. We at deepsense.ai are not happy with this approach and strive to provide solutions that actually ARE better.

Technical causes: Why does EDS plague GenAI?

Technically speaking, why exactly would EDS haunt GenAI applications? We figured out the Traditional ML evaluation quite well. Why would this know-how not translate easily to GenAI? In this section, we’ll delve into the ‘hard’/technical factors behind EDS in Generative AI.

Let’s start by refreshing the (very straightforward) conventional approach to ML development and evaluation. Let’s break it down into three steps:

1. gather human-generated ground truths (GTs),
2. gather model-generated outputs,
3. compare them directly to produce meaningful metrics like accuracy, F1 score, and mean squared error.

There are several reasons why this approach breaks down for GenAI, regardless if it is related to the generation of text, images, or other content.

1. Inadequacy of the ‘ground truth’ concept in GenAI

Firstly, almost by definition, generative tasks often include a practically infinite variety of ‘optimal’ solutions, making any metrics which rely on direct comparisons with GT questionable. Within Natural Language Processing (NLP), ‘pseudo-evaluation’ approaches that we call ‘Superficial Utility Comparison Kriterion’ (SUCK) methods, like BLEU [32], METEOR [33], ROUGE [34], or BLEURT [35], attempt to salvage the situation. SUCK methods usually compare model outputs to GTs in specific embedding spaces, de facto under the assumption that output quality correlates with similarity to some GT within that space.

While this assumption might hold to some extent in quasi-generative tasks like summarization [link], it is fundamentally flawed. SUCK’s weaknesses are well recognized in the literature [link]. In our opinion, a fundamentally different approach is needed – we simply need to embrace the fact that ‘ground-truth’ is simply not a viable concept for most truly generative tasks. When the task is love poem generation, what is THE right answer that all others should imitate?

2. The innate subjectivity in GenAI evaluation

Secondly, assessing human-level creativity requires subjective and elusive criteria. Such evaluation is best expressed by terms like aesthetics, novelty, style, creativity, helpfulness, and poorly-defined concepts like accuracy. Formal quality formulas have limited promise in this context. Furthermore, intersubjectivity is an inherent feature of evaluation, not a flaw to eliminate [36]. In many cases, generative models should optimize multiple objective quality indicators (e.g., the correctness of the generated code or retrieval ratio correctness etc.) and the subjective ones (e.g., code cleanliness, or proper conversational tone). All of them may non-trivially determine the final output quality, and more often than not conflict in practice [37, 38], and how to use them all in the practical evaluation is not necessarily obvious. Data collection, training, and evaluation procedures need to explicitly account for, quantify, and control subjectivity in the subdimensions of GenAI evaluation and their relation to the final quality of the output.

3. Extreme use-case specificity of the GenAI evaluation criteria

Thirdly, the interpretation of values relevant to the problem is highly application-specific and changes dramatically between use cases, and even users. Despite using identical terms, domains or use-cases dramatically redefine their interpretation. Context and goals determine standards of output beauty/aesthetics, proper tone, relevance, social acceptability or most other subjective metrics one can imagine. This makes capturing them with large, cross-domain, ‘once and for all’ datasets a challenge.

4. Diversity / mode collapse monitoring problems

The fourth reason is the susceptibility to mode collapse, where models produce outputs with limited diversity. While this can impact the utility of the model, it is also difficult to quantify and measure as part of model evaluation. Attempts to approximate the human sense of diversity, like FiD or the CLIP score, have serious limitations [39, 40]. On one hand, the definition of sample proximity may be non-trivial and domain-specific. On the other hand, diversity may be hard for humans to evaluate, as it is a feature of a large dataset – evaluation of multiple objects is a natural human weakness.

5. Potential evaluation dataset leaks in behind-closed-doors training

The fifth and final challenge in GenAI evaluation is the potential data leakage of any existing evaluation datasets one intends to utilize. Almost any model trained by the GPU-poor is a refinement of a general model (like Llama or Stable Diffusion) generously thrown open by the GPU-rich. These models have been trained on just about any data available, including all open datasets you wish to use to evaluate them or their fine-tuned iterations. The closed-door nature of many of these training procedures and their often undisclosed training datasets complicate the issue further. When we consider this in conjunction with the third challenge, which relates to the high degree of application specificity in given metrics, it becomes evident that depending on vast, openly accessible, one-size-fits-all datasets may no longer be a viable approach. It seems that the GPU-poor may have to rely on methodology allowing for easy evaluation with internally created, very small datasets capturing the case-specificity of their application.

Okay, so there are technical reasons why GenAI evaluation is hard, and why traditional ML approaches fail. But why would those technical obstacles hit the GPU-poor more? Why and how are the GPU-rich capable of dealing with them? Why can’t the GPU-poor copy what they do on a small scale? These are all great questions that we will address next.

How do the rich kids do it? The shiny new HFRL you (probably) can’t afford

Time for full disclosure: the problem above is only partially new. A lot of thought went into the proper approach to somewhat similar challenges long before the GenAI revolution. The field of Reinforcement Learning (RL) has much to say about the ‘healthy’ approach to a situation where the evaluation metric (‘reward’ in the RL lingo) is hard to formalize, complex, etc.

Successful RL conceptual frameworks like actor-critic [41] or world-model [42] have been used in physical or virtual environments to attack problems with ill-defined notions of ‘correct action’. The moment one sees GenAI as an agent, its outputs as actions, and human evaluation as a reward function, it all falls very nicely into the RL framework – hence Human Feedback Reinforcement Learning (HFRL). One RL-borrowed component that allowed the GPU-rich to ‘put the harness’ on the GenAI they produced was the notion of a critic, preference, or reward model. Understanding this concept is vital to our conversation. The preference model is trained on human evaluations, to approximate human, well… preferences (Image 3, left). Once successfully trained, it can provide automated, pseudo-human feedback to train the generator (Image 3, right) concerning the human-level metrics (friendliness, political correctness, etc.). It is relevant to note that the reward model is a traditional (i.e., non-generative) model – it is not GenAI, so we know how to evaluate it directly. OpenAI used this approach to make ChatGPT by ensuring that GPT3/4 will not set off on racist rants, etc., by finetuning it to score highly on a reward model trained for OpenAI-preferred ideological biases. However, the same principle works for any GenAI (for example, reward models trained on human aesthetics [43] enable the creation of image generators like Midjourney [44]).

It should now be clear that HFRL makes the GPU-rich immune to EDS. Preference models can provide an automated, repetitive, quantitative evaluation for human-level metrics like beauty or friendliness. There is a ‘but’, however. Namely, the way GPU-rich use HFRL is categorically beyond the rich of GPU-poor. We cannot ‘just copy what they do on a smaller scale’ – we are too poor to do even that. The amount of GPU power, labeling, and the (often overlooked) scale of technical engineering difficulty needed to make full HFRL work is unknown and (probably rightfully) normally assumed prohibitive for today’s GPU-poor.

At this point, many highly technical GPU-poor experts, who intuitively feel much of what we have described, just shrug and say, ‘that’s why we cannot afford proper GenAI evaluation’. One reason for this may be a misleading intuition about what is hard in (HF)RL. The thing is, the difficulty considered here is not just training any evaluation model, but rather a reward model strong enough to be used in full HFRL, i.e., the provider of a training signal directly from the generative model (right column). In this setup, the reward model becomes vulnerable, as the generative model may ‘beat’ it, exposing any weakness (hidden misalignments with human preferences) it may have. Dealing with this has much in common with adversarial training stability challenges in GANs [45] or general issues with ML adversarial safety [46]. It is hard. In essence, the reward model and the entire training methodology must be very strong to stop the generator from ‘hacking’ the preferences, or it will find a way to get a high reward in undesired ways (by generating weird or noise-like images/sentences that should not get high rewards, but do). This takes a vast amount of data, GPUs, and expert engineering.

Here comes the trivial observation behind the Evaluation Driven Development (EDD) that the GPU-poor can adapt. Evaluation models may be easier by orders of magnitude to obtain than reward models. Our experiments indicate that evaluation models may be very cheap and easy to create for the GPU-poor’s own use cases, when taken out of the HFRL context. Unlike the GPU-rich shaping their base models, you do not need your reward model to provide a training signal for the model you fine-tune (or prompt engineer). If you do it right, and the generator cannot fool the evaluator through the training signal, as few as 100-200 samples may be enough to create human-like evaluator models. And that is all you need to get out of the EDS!

Okay. So we have the general idea behind the EDD. But that’s not really a framework yet, is it? And what does ‘if you do it right’ mean exactly? That’s an entirely different story – one we will tell in the second blog post of this series.

References

1. A survey of Generative AI Applications, Gozalo-Brizuela R., Garrido-Merchán E.C., 2023
2. From ChatGPT to ThreatGPT: Impact of generative AI in cybersecurity and privacy, Gupta M., Akiri C., Aryal K. et al., 2023
3. Art and the science of generative AI: A deeper dive, Epstein Z., Hertzmann A., Herman L. et al., 2023
4. https://www.businessinsider.com/gpu-rich-vs-gpu-poor-tech-companies-in-each-group-2023-8?IR=T
5. https://www.semianalysis.com/p/google-gemini-eats-the-world-gemini
6. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
7. BERTScore: Evaluating text generation with BERT, Zhang T., Kishore V., Wu F. et al., 2019.
8. BLEU: A method for automatic evaluation of machine translation, Papineni K., Roukos S., Ward T. et al., 2002.
9. https://github.com/features/copilot
10. USR: An unsupervised and reference free evaluation metric for dialog generation, Mehri S., Eskanazi M., 2020
11. Better automatic evaluation of open-domain dialogue systems with contextualized embeddings, Ghazarian S., Wei J. T.-Z., Galstyan A. et al., 2019
12. An empirical study on evaluation metrics of generative adversarial networks, Xu Q., Huang G., Yuan Y. et al., 2018
13. Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models, Stein G., Cresswell J.C., Hosseinzadeh R. et al., 2023
14. Witscript 3: A hybrid AI system for improvising jokes in a conversation, Toplyn J., 2023
15. MaxProb: Controllable story generation from storyline, Vychegzhanin S., Kotelnikova A., Sergeev A., Kotelnikov E., 2023
16. Modern French poetry generation with RoBERTa and GPT-2, Hämäläinen M., Alnajjar K., Poibeau T., 2022
17. SDXL: Improving latent diffusion models for high-resolution image synthesis, Podell D., English Z., Lacey K., 2023.
18. https://www.gartner.com/en/articles/what-s-new-in-artificial-intelligence-from-the-2022-gartner-hype-cycle
19. https://pitchbook.com/news/articles/generative-ai-hype-stability-jasper
20. https://www.wsj.com/articles/no-business-plan-no-problem-chatgpt-spawns-an-investor-gold-rush-in-ai-6bdbed3c
21. https://www.cfodive.com/news/many-ai-early-adopters-struggle-success/652138/
22. https://mergers.whitecase.com/highlights/generative-ai-boom-sparks-investment-spree#!
23. https://techcrunch.com/2023/03/28/generative-ai-venture-capital/
24. https://sifted.eu/articles/keelvar-ai-go-to-market
25. https://ai.meta.com/llama/
26. https://mistral.ai/news/announcing-mistral-7b/
27. https://zapier.com/blog/llama-meta/
28. https://www.digitaltrends.com/computing/why-meta-llama-2-is-such-a-big-deal/
29. https://www.techopedia.com/definition/google-gemini
30. https://medium.com/@trendingAI/gpt-5-revealed-release-date-what-we-know-so-far-ff7ec532d40e
31. https://www.zenger.news/2023/06/10/sam-altman-says-its-hopeless-to-compete-with-openai/
32. BLEU: A method for automatic evaluation of machine translation, Papineni K., Roukos S., Ward T. et al., 2002
33. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments, Banerjee S., Lavie A., 2005
34. ROUGE: A package for automatic evaluation of summaries, Lin C.-Y., 2004
35. BLEURT: Learning robust metrics for text generation, Sellam T., Das D., Parikh A.P., 2020
36. Understanding interobserver agreement: The kappa statistic, Viera A.J., Garrett J.M., 2005
37. Improved precision and recall metric for assessing generative models, Kynkäänniemi T., Karras T., Laine S., 2019
38. Evaluating creative language generation: The case of rap lyric ghostwriting, Potash P., Romanov A., Rumshisky A., 2018
39. Toward verifiable and reproducible human evaluation for text-to-image generation, Otani M., Togashi R., Sawai Y. et al., 2023
40. Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models, Stein G., Cresswell J.C., Hosseinzadeh R. et al., 2023
41. https://medium.com/intro-to-artificial-intelligence/the-actor-critic-reinforcement-learning-algorithm-c8095a655c14
42. Deep learning, reinforcement learning, and world models, Matsuo Y., LeCun Y., Sahani M., 2022
43. https://laion.ai/blog/laion-aesthetics/
44. https://www.midjourney.com/showcase/recent/
45. https://jonathan-hui.medium.com/gan-why-it-is-so-hard-to-train-generative-advisory-networks-819a86b3750b
46. Key concepts in AI safety: Robustness and adversarial examples, Rudner T.G.J., Toner H., 2021

A comprehensive review of the state of the art in terms of code-writing agents. We compare the existing solutions and explain how they work behind the scenes.

Introduction

While the concept of AI agents has been around for decades, it is undeniable that recent advancements in Large Language Models (LLMs) have revolutionized this field, opening up a whole new realm of possibilities and applications.

This article will walk you through the current landscape of code-writing AI agents. We present a couple of the existing implementations and discuss how they are built.

In a follow-up article (stay tuned) we’ll share our experiences with developing our own version of an AI agent. We’ll also compare open-source agents and our own ones based on the quality of code written, cost, usage, and speed.

What is an AI Agent?

Imagine having an ever-present, intelligent companion who is always at your side, ready to lend a helping hand whenever you ask, and who never says ‘no’. The AI agent of the future will be exactly that.

We can define an AI Agent as a computer program or system that can perceive its environment, process information, and make decisions or take actions to achieve specific goals (such as solving software engineering problems). AI agents are designed to mimic some aspects of human intelligence and behavior, allowing them to operate autonomously or semi-autonomously in their environment.

Traditionally, AI agents were primarily utilized in the domain of game development. From simple chess-playing programs to complex virtual characters with human-like behaviors, these agents have showcased impressive capabilities within constrained environments. However, the limitations of traditional AI methods often restricted their adaptability, creativity, and real-world applicability.

Usually agents will have:

• Some kind of memory (state)
• Multiple specialized roles:
  • Planner – to “think” and generate a plan (if steps are not predefined)
  • Executor – to “act” by executing the plan using specific tools
  • Feedback provider – to assess the quality of the execution by means of auto-reflection. It can be augmented or replaced by human feedback.
• External sources of informationA way to interact with the user (an optional feature)

Attempts have been made to create agents in various fields such as:

• Robotics: Agents can range from simple automated machines to complex humanoid robots designed to assist or replace humans in various tasks.
• Autonomous vehicles: These agents have the potential to revolutionize industries, especially transportation, in the form of self-driving cars, trucks, and drones.
• Gaming: The use of intelligent agents in the development of more immersive and challenging gaming experiences is on the rise.

A significant aspect of these intelligent agents is their connection with Reinforcement Learning. Reinforcement Learning is one of the seminal branches of Machine Learning, with roots reaching back to the 1980s and 1990s. Despite its age, it remains at the bleeding edge of technology, driving the development of increasingly intelligent, adaptable and complex systems.

Although LLM-based agents are relatively new (as, in fact, are LLMs themselves), they are already being applied to a wide range of tasks, such as:

• Helping you search through long documents and asking them questions: How we developed a GPT-based solution for extracting knowledge from documents – deepsense.ai
• Playing games: GitHub – MineDojo/Voyager: An Open-Ended Embodied Agent with Large Language Models
• Creating entire software engineering projects for a given problem statement: GitHub – AntonOsika/gpt-engineer: Specify what you want it to build, the AI asks for clarification, and then builds it.

In this article we will focus solely on AI Agents capable of solving software engineering and data science problems.

The current state of coding agents

LLMs have rapidly revolutionized the code creation process as developers eagerly include them in their workflow. Whether programmers ask ChatGPT their questions directly and make use of the code snippets it provides, or use dedicated code-writing tools like Copilot which are capable of suggesting whole code blocks at a time from the convenience of your IDE, one thing is certain: the utilization of AI-powered tools in our daily work provides a significant productivity boost. The burning question of today is whether we can take this idea even further and step away from code suggestions, moving towards end-to-end solutions capable of creating the entire project by themselves. AI Code Writing Agents represent a promising approach to this task.

There are a number of open-source projects introducing fresh approaches to AI Agent implementation. Among them, a couple stand out with their relative maturity and serve as inspiration for any new projects in this field.

General purpose coding agents

Auto-GPT

Auto-GPT was one of the first AI agents using Large Language Models to make waves, mainly due to its ability to independently handle diverse tasks. The AutoGPT agent has a large tool set of commands it can call upon, including:

• Web-based functions like Google searches and browsing
• Interactions with fellow agents (initiating, messaging, deleting)
• Operating system operations like file manipulation and script execution

Workflow

Operating via a chain-of-thought feedback loop, the agent generates prompts that it loops back into its own processing. Let’s look into the workflow following an example – creating a sentiment classifier.

1. The user provides a prompt setting the task.
2. An agent is created based on the prompt.
   • The agent has a personality specified – a SentimentClassifierGPT – making it well prepared for the task at hand.
3. The agent thinks about how best to solve the problem.
   • To make sure there’s no jumping to conclusions, it is bound by a deliberate thinking structure – the chain of thought.
4. The agent selects a command to run, and comes up with arguments to this command.
   • This time it went with executing python code, providing the code it came up with as an argument.
5. Time for us, the user, to authorize the action.
   • Silly agent – it doesn’t know that running this code is a bit premature! Let’s roll with it anyhow – accept.
6. The result of running this command gets memorized.
   • If we have more rounds in our tank, the workflow goes back to the thinking step, and the agent gets to reflect on the results of its actions.
7. Second round! Workflow loops back to step 3.
   • The agent thinks – “The error indicates that the ‘torch’ module is not installed in the current environment… However, as an AI, I don’t have the capability to install packages.” Spot on, buddy! We are not going to succeed this particular time.

Extra details

For memory, the agent employs a dual approach. It possesses a short-term memory in text format, complemented by long-term memory embeddings within a vector database. This enables the agent to effectively leverage contextual information and the outcomes of past actions within its ongoing interactions.

Pros of Auto-GPT

• Versatility – an agent can call up a large variety of commands to help it with the task at hand.
• Chain-of-thought prompting – a brilliant technique helping agents hit the mark more often.
• Memory handling – well-designed short- and long-term memory.

Cons of Auto-GPT

• Operating within a loop – the costs can rack up quickly! Especially when the agent exhibits repetitive behavior.
  • This is slightly evened out by the human control mechanism – the user can stop the loop if it seems to be going nowhere.
• Memory handling and context limitations – while it is well designed, it is still one of the weakest chains in this tool.
  • For big projects, agents are prone to forget what they have already accomplished, which makes them likely to repeat themselves.
  • The relatively small context of the current LLMs makes memory handling even more challenging.
• Poor code-writing abilities.

What makes the project stand out

• Agents created on the fly

BabyCoder

BabyAGI is another AI agent working in a loop. It has inspired multiple projects, possibly due to the simplicity of its implementation.

BabyCoder is one of the BabyAGI variants which specializes in writing code. Unlike AutoGPT, which creates agents on the fly, BabyCoder uses a handful of predefined, specific agents who work together to come up with a functional project.

Workflow

1. The Task Initializer Agent creates a list of tasks to be carried out,
2. The Human Input Agent works with the user to refine the tasks
3. The Task Assigner Agent passes each task to a capable executor, such as
   • the Command Executor Agent
   • the Code Writer Agent
   • the Code Refactor Agent

Let’s try this out on the same task – creating a sentiment analysis model.

1. The user provides a prompt setting the goal.
2. The task Creation Agent comes up with tasks – in our case all 14 of them. The tasks span from creating project and data directories, to installing the required python modules, to writing the specific code files.
3. The Command Executor Agent starts working on the first task – and performs it flawlessly: mkdir sentiment_classifier/data
4. The context is extended by the result of the first task.
5. The execution agent picks up the next task on the list.
6. Workflow loops back to step 3 – task execution.
7. … The program crashes.

While most likely not the intended workflow, that is exactly how this turned out during our tests. At this point the BabyCoder either forgets what directories it created and attempts to use incorrect paths, looping itself aimlessly, or exceeds the available LLM context window with a loud error message straight from the OpenAI API.

Splitting the main task into multiple small, specialized tasks is a brilliant workaround for the current limitations of LLMs, including their limited attention span and difficulty handling complex workflow.

Pros of BabyCoder

• Very simple implementation

Cons of BabyCoder

• Unreliable – tends to crash with an error before finishing the tasks it set

What makes the project stand out

• Predefined specializing agents
• The number of projects it has inspired

smol developer

smol developer lives up to its name by being lightweight and simple.

Workflow

1. The user initiates the app-building process with a singular prompt.
2. The smol developer creates a detailed plan that outlines the structure of the app to be generated, listing all the variables and functions. This plan will be included in each prompt the smol developer uses later on.
3. The agent creates the code according to the first task.
4. Memory is extended by the code generated.
5. The agent creates the code according to the second task… And so on.

Pros of smol developer

• Simplicity
• Linear workflow – no surprises in terms of the number of calls made to the OpenAI API

Cons of smol developer

• Lack of advanced memory handing – smol developer stuffs all the information together, so it needs to fit inside the LLM context
• Requires multiple tries before you get the prompt right

What makes the project stand out

• Simplicity

GPT Engineer

GPT Engineer is a tad bigger, but still a lightweight approach to code generation. It starts from a prompt – but it also asks extra clarifying questions.

Workflow

1. The user provides the main objective.
2. The GPT Engineer asks clarifying questions about the target, focusing on the technical aspects of the solution (e.g., data format, preference for the model used). The user can answer or can leave it to the model to decide by itself
3. The agent prepares the “Core Classes, Functions, and Methods” section, which lists technical details of the solution. For example 1. `SentimentDataModule`: A PyTorch Lightning DataModule for loading and preparing the sentiment analysis dataset.
4. Based on the code functionalities described above, the GPT Engineer starts to create code, one element after another
5. Once the code is generated, the agent gives us the option to install the necessary libraries (from the requirements.txt file) and launch his creation

It seems that the code produced by gpt-engineer is already much better than that produced by its predecessors, although it is still far from perfect.

Pros of GPT Engineer

• Clarifying questions – the agent asks questions to make sure it produces code that is consistent with our expectations
• The workflow is concise, as is the code itself, which saves us time and money
• Gives you the ability to install dependencies and execute code

Cons of GPT Engineer

• The questions do not always make sense – sometimes they ask about information already included with the initial prompt

What makes the project stand out

• Objective clarification
• Dependency handling and code execution
• Alternative approaches to code generation, for example Test-Driven Development (TDD)
• At the end of the execution, the GPT Engineer collects data on the quality of its work – signaling the authors’ intention to keep working on the improvements

MetaGPT

MetaGPT originated as a research paper, with its focus centered on multi-agent collaboration.

This project is heavily inspired by how human developers manage IT projects. MetaGPT is written in a way that resembles a software company, hiring agents for positions common in the industry – like architects, product managers and even customer service reps. MetaGPT is a complex tool, which is why we decided to describe it in more depth than usual.

Workflow

1. The user provides the prompt.
2. The boss agent broadcasts the goal.
3. The agents check the shared message board for information relevant to them.
4. If an agent sees something of interest it reacts.
   1. Think – keeping in mind the context of the role, see what needs to be done and reflect on how to do it.
   2. Observe – check the message board and read about the relevant events.
   3. Act – perform a task, call an LLM or another function.
   4. Broadcast – share the execution results and other action records with the environment.
5. After performing their actions, agents broadcast their results to the message board.
6. Back to point 2.

The description above, while accurate, doesn’t do much in terms of explaining what actually happens. For a step-by-step walkthrough, see below:

Example of an actual workflow

Portrayed using the sentiment analysis model creation prompt.

• The user provides the prompt.
• The boss agent picks up the prompt and passes it on to the company message board.
• All of the employee agents look through the message board, looking for any information of interest.
• Alice, the Product Manager, picks up the requirements and gets to work:
  • Polishing the requirements
    • “Develop a sentiment classifier based on product reviews and their ratings” (…)
  • Creating the user stories
    • “As a data scientist, I want to train a sentiment classifier so that I can predict the sentiment of product reviews” (…)
  • Drafting the UI
    • “The scripts should be command-line executable and provide clear output to the user.” (…)
  • Performing the competitive analysis by searching for the information on the internet
    • “Google’s AutoML Natural Language: Offers sentiment analysis but lacks customization for specific tasks” (…)
  • Documenting her findings using mermaid charts (see figure 3)
  • Posting the results of her work to the company message board
• Bob, the Architect, sees that Alice is done and picks up the baton.
  • Drafting the implementation approach
    • “We will use the transformers library from Hugging Face, which provides pre-trained BERT models and utilities for fine-tuning on a specific task.” (…)
  • Deciding on the python package name
    • “sentiment_classifier”
  • Creating the file list
    • “train.py”,
    • “validate.py”,
    • “inference.py”,
    • “model.py”,
    • “utils.py”
  • Charting the data structures and interface definitions (see figure 3)
  • Posting the results of his work to the company message board
• Eve, the Project Manager, details the tasks
  • Analyzing the dependencies
    • Creating the requirements.txt
  • Creating full API specification
  • Analyzing the program flow
    • “train.py”, “Contains the main training loop. Depends on model.py and utils.py”
  • Posting the results of her work to the company message board
• Alex, the Engineer, writes the code for each file.

Environment

The agents share an Environment where all the memories are stored. Memories are essentially a list of messages broadcasted by agents. Agents with specific roles can browse the environment to retrieve and store memories relevant to them. Each agent possesses a distinct context, subscribing to different types of events and striving to remember only what is directly relevant to its role.

Working in rounds

The execution of the program is limited by the number of rounds set, so it’s important to leave enough rounds for every agent to have an opportunity to act.

For example, during the first round, the only message in the environment comes from the boss – that’s the initial user prompt. The Project Manager will pick up the requirements, perform her actions and post the results to the environment message buffer. However, no other agent will pick up this information during the same round – they have already finished their observations. Starting next round, Bob, the Architect, will pick up the user stories and start coming up with the design. A couple of rounds have to pass before any code gets written.

Each agent can react any time an event of interest is registered in the environment.

In some cases multiple agents will take action during the same round – a state management system has been implemented to prevent conflicts between asynchronous actions, allowing an agent to indicate whether it is busy or idle. This design might make it easier to develop this solution towards multithreading – but at the moment it does not support parallelism.

The mermaid charts

MetaGPT outputs very detailed schematics of its designs using the mermaid diagramming tool.

Pros of MetaGPT

• Standardized Operating Procedures – well-designed workflow reduces chaos and ensures the tasks are handled consistently
  • The charts!

• High quality solutions
• A budget cap – a spending limit keeping us safe

Cons of MetaGPT

• Not ideal for simple tasks – the complex workflow can be an overkill

What makes the project stand out

• Complexity – in terms of tools, memory handling and overall workflow
• Academic background – an excellent paper describes MetaGPT in detail

Comparing the models

	AutoGPT	BabyCoder	smol developer	GPT Engineer	MetaGPT
Workflow	Loop	Linear	Linear	Linear	Loop
Order of actions	Sequential	Sequential	Sequential	Sequential	Asynchronous
Purpose	Multipurpose	Coding	Coding	Coding	Coding
Memory	Selectively retrieved	Selectively retrieved	Context only	Context only	Selectively retrieved
Agents	Created on the fly	Predefined	Predefined	Predefined	Predefined

Table 1. Comparison of the discussed models, Source: own study

None of the models we ran produced a working sentiment analysis model. AutoGPT and BabyCoder were not able to finish their processing without errors. GPT Engineer and MetaGPT came closest to a working solution. MetaGPT impressed us the most due to the extras it produces – charts and specifications. Of the two, it was also easier to use – GPT Engineer takes more effort to start due to the extra clarifying questions it asks at the beginning.

Specialized coding agents

DemoGPT

DemoGPT allows users to create interactive Streamlit apps with just a prompt. What makes DemoGPT different from other solutions is that it specializes in generating LLM-based applications powered by Streamlit and LangChain as opposed to more generic agents described above.

The core functionalities of DemoGPT revolve around four distinct stages (see Figure 4):

1. Planning: DemoGPT generates a comprehensive plan based on the user’s instructions.
2. Task Creation: Next, specific tasks are created by DemoGPT, utilizing the generated plan and the provided instruction as a guide.
3. Code Snippet Generation: The generated tasks are then transformed into code snippets.
4. Final Code Assembly: The code snippets are merged to form a final code, resulting in an interactive Streamlit app.

Moreover, each of these steps undergoes refinement before progressing to the next stage, ensuring optimal quality and accuracy throughout the process. Once the user approves the final outcome, all the generated artifacts, including the plan, tasks, and code snippets, are saved in the database for future retrieval when generating similar applications.

By utilizing these functionalities, DemoGPT is able to convert user instructions into interactive applications, making it an effective tool for LLM-based application development.

Wasp

At its core, Wasp tries to redefine full-stack web application development. It covers three significant domains of a web application, creating a well-functioning client (front-end), server (back-end), and the database, all connected in one ecosystem.

Underneath, Wasp is fueled by React, Node.js, and Prisma. With these technologies at its backbone, Wasp defines web components, server queries, and actions.

The essence of its effectiveness is Wasp’s built-in compiler. Thanks to this, you only need to describe your features in a special config file, and Wasp will compile them for you. The output will be the client app, server app and the whole codebase needed for deployment.

Everything described above is summarized in the figure below:

And this is what a very basic sample Wasp config file looks like:

You may wonder how the concept of generative AI tools comes into play here when we are the ones writing the code and configuration file. Well, Wasp has been on the market for quite some time. However, its team just recently started integrating GPT to develop a full web application with only a natural language objective specification.

The model does a great job of creating Wasp configurations and individual components. It facilitates the quick construction of a simple application prototype, complete with a database, an authentication system, and the backbone of essential CRUD operations.

You can test Wasp’s capabilities in generating web applications from scratch here, where a free playground has been provided.

The future of AI Agents

We’re still in the early stages of LLM-powered AI Agent development, but they already have impressive capabilities. We believe that in the next few years we will see rapid improvements in that field and rapid increase in the products based on AI Agents. These solutions will include not just language-based models (LLMs), but also multimodal models, among others – allowing agents to work with different types of data like images and videos. Frankly, we already have models like GPT4 and Bard that are multimodal and can take images as an input, but it seems that these capabilities are not yet fully utilized in AI Agents.

We may even ask ourselves: can these AI agents lead to AGI?

Summary

In this blog post, we’ve discussed the rapid growth of AI Agents, fueled by recent breakthroughs in LLM. These AI agents attempt to write complete code for projects simply from the problem statements – albeit not always successfully. There are a variety of solutions offering different approaches to code writing – from the broad capabilities of MetaGPT, to the specialized nature of Wasp. However, every one of them shares a similar core – an LLM answering questions and creating code – and they can only be as good as the underlying model.

Currently, these solutions are far from being a replacement for humans – but if we give them time, they may come to surprise us all.

If you are interested in evaluating these agents and the ones we have created, as well as the lessons we learned from creating AI support for writing code, stay tuned for the upcoming second part of our blog post!

Need a reliable partner for a forward-thinking AI project for your business? Contact us today, and our experts will help you find a bespoke solution tailored to answer your business challenges. Choose our AI development services!

The recent strides made in the field of machine learning have given us an array of powerful language models and algorithms. These models offer tremendous potential but also bring a unique set of challenges when it comes to building large-scale ML projects. In this blog post we will discuss the importance of LLMOps principles and best practices, which will enable you to take your existing or new machine learning projects to the next level.

Naturally, training a machine learning model (regardless of the problem being solved or the particular model architecture that was chosen) is a key part of every ML project. Preceded by data analysis and feature engineering, a model is trained and ready to be productionized. But what happens next?

We may observe a growing awareness among machine learning and data science practitioners of the crucial role played by pre- and post-training activities. Assigning more weight to these parts is one of the key principles of LLMOps.

What is LLMOps?

To start simply, you could think of LLMOps (Large Language Model Operations) as a way to make machine learning work better in the real world over a long period of time. As previously mentioned: model training is only part of what machine learning teams deal with. Other steps include:

• data ingestion, validation and preprocessing,
• model deployment and versioning of model artifacts,
• live monitoring of large language models in a production environment,
• monitoring the quality of deployed models and potentially retraining them.

Of course, in the context of Large Language Models, we often talk about just fine tuning, few-shot learning or just prompt engineering instead of a full training procedure. Nevertheless, this also requires certain steps such as the preparation of prompt templates, evaluation of results etc. Moreover, however you decide to improve the accuracy of your models, you still need to make sure your system works reliably and consistently.

As a set of concepts, guidelines and tools, LLMOps helps with all of these steps by providing tools and practices that make it easier to manage and maintain a machine learning system. Those tools and practices not only help to integrate consecutive steps (see Figure 1) together and make them work smoothly; they also make sure that the whole process is reproducible, automated and properly monitored at each stage – model training as well as model inference. This approach is as important for fine-tuning Large Language Models internally as for projects where you only use LLM-as-a-service for the inference part (think of, e.g., using OpenAI API for calling GPT models). Even though you don’t train or own the model, you still should pay attention to what data comes in and out of it. One should carefully monitor the whole process, log the predictions and so on.

Now that we have a common understanding of the term itself, let’s delve into how adopting LLMOps principles will make your project more efficient, sustainable and robust.

Making sure your LLMs are healthy and produce the desired results

One of the most important concepts, the importance of which I cannot stress enough, is monitoring. It covers monitoring the quality of input and output data, overall responsiveness and behavior of models as well as observing the traffic that goes into your LLM-based application. Why are these elements so important?

You have probably already invested a great deal of time in choosing the right architecture for the application, the whole team has decided which large language model would be the best choice for your use case, the model has been fine-tuned on your knowledge base, you have all the pieces and you are ready to go into production. That’s great! However, what usually happens in such scenarios is that right after a model is deployed in production, teams are gradually reassigned to other projects, analyzing new data and training new models. But the application is still live, right?

A lot of things may go wrong from here, e.g.:

• your system began to receive much more traffic than you initially estimated and is not able to handle so many requests (see Figure 2), resulting in many errors being returned and users being unsatisfied with your product,
• even if you don’t have a problem with the stability of the solution, the response time may turn out to be unacceptable for end users, as they have to wait too long for the responses due to the large amount of traffic and bottlenecks in your application,
• production traffic is very inconsistent, resulting in huge peaks of user requests during which your system consumes a large number of resources (RAM, CPU or GPU utilization, as in Figure 2). For a certain period of time it may not cause any trouble, but with more and more traffic it may eventually lead to the services hosting your Large Language Models frequently crashing.

There are more challenges than those listed above, such as changes in model behavior due to a sudden drift of input data (see Figure 3). When users start to use your application differently than you expected, the predictions of Large Language Models can also change significantly. This can lead to incorrect results being presented to end users.

Implementing a proper monitoring platform, dashboards visualizing resource consumption and application health is essential. We cannot forget about other components that are part of modern applications that utilize Large Language Models, e.g., vector databases, frontend services – every part of a mature, production-grade system should be carefully monitored. Naturally this applies to every long-term LLM-based application, whether it is a chatbot, a simple knowledge retrieval service or any other use case.

Allowing engineers to do productive work instead of tedious tasks

Apart from monitoring, another key concept of LLMOps is the automation of software development (or machine learning) processes. In many projects, a lot of time and effort is assigned to exhausting and repetitive work such as:

• the manual deployment process, requiring engineers to go through a checklist of actions to be able to push a newer release of the application into production; even scarier is that in case a bug is found, such a process has to be manually reversed step by step,
• regularly checking the monitoring dashboard to see if the application works as expected, producing the desired results (as discussed in the previous section) etc.

The problem is that, despite being so monotonous, these tasks are at the same time very important for your application and also highly error-prone! Not the best combination, right?

What we can do about this problem is to automate these tasks (see Figure 4) to make sure they are executed frequently in the same manner, leaving room for human error and allowing developers to focus on other tasks at the same time, increasing your team’s productivity.

The manual deployment process can be replaced with an automated continuous delivery pipeline which takes care of testing your LLMs and application thoroughly. Developers may only be needed to finally approve the deployment process but not to execute a whole list of manual steps. Instead of regular, tedious checking of charts and values, you could configure a monitoring platform to send you a notification (it could be Slack, e-mail or other tools you use) whenever, e.g., a metric drops below a certain threshold or an application becomes unresponsive for a certain amount of time. Take a look at an example of such a setup presented in Figure 4. In such a setup there are only two manual actions where a human-in-the-loop is necessary while the rest happens without human intervention:

• a Data Scientist merges a change in the training code or parameters to a codebase. This triggers a bunch of quality checks (e.g. linting, tests) and starts a training job. During training, we log all the model metrics and metadata automatically. A successful run produces model artifacts saved in a model registry, ready to be picked up for inference,
• in the second part, the team validates model metrics and decides to deploy the model they have just trained. A Machine Learning Engineer approves a model in the registry and this action automatically triggers a validation procedure which, if successful, delivers a working deployment into an inference environment.

Of course, the desired level of automation is different for each project. It depends on the scale, the frequency of deployments, and the overall maturity of the machine learning system. But this example should give you an overview of how properly applied automation processes can reduce the risk of human error and make the whole end-to-end process much faster and more reliable.

Reducing the overall cost of the project and application

Last but not least, implementing an LLMOps platform can eventually lead to significant savings in your project budget. Let’s discuss this further with an example of the aspects we described earlier (i.e., monitoring and automation).

First of all, thanks to monitoring, you will know the exact utilization of your infrastructure. If your LLM-based service is deployed in a cloud environment, for example, you can adjust the type of instance you are using to avoid unnecessary costs for resources you do not consume. On the other hand, if you use external models as an API (such as OpenAI GPT), it may be a good idea to track your expenses using a billing dashboard provided by the vendor or to build an in-house system to observe the costs and their correlation with, e.g., the length of input being sent to these models.

Observing such metrics will allow you to compare different architectural choices, cloud providers or specific Large Language Models that you deploy. Monitoring is crucial, not only for the sake of keeping your project healthy but also for optimizing business metrics and expenditure.

On the other hand, as we already mentioned, automation can free up your team from working on monotonous tasks on a daily basis and will allow you to utilize their knowledge and time in a much more productive way.

There are also other LLMOps concepts or practices that we did not explicitly mention in this blog post, such as model retraining and automatic data labeling. More than that, applications that utilize Large Language Models often include other key components such as vector databases – exactly the same principles (of monitoring, automation etc.) apply to these parts of your system.

All of these, when implemented correctly, can help you maintain your project in a more robust and cost-effective way.

Conclusion

Having LLMOps (or MLOps, in the context of smaller language models) best practices implemented in your machine learning system makes it more reliable, robust and automated. Once such standard practices and automation are implemented, delivering new models into production will require nothing but a few commands or clicks, as the rest of the process will happen behind the scenes. You will be able to track every log, metric and behavior, and thanks to the monitoring stack you will be able to react to events or unforeseen behaviors quickly.

And once again, keep in mind that using models-as-a-service through third party APIs from OpenAI or other platforms does not mean that you do not need to apply LLMOps principles anymore. In fact you only outsource the training and model-serving parts of the project and while it may not be your concern for short-term MVP projects, you should definitely look at LLMOps-related practices whenever you plan to build a serious machine learning-based product.

Diffusion models operate in a generative capacity that empowers them to create images using varying data sources. The resultant outputs are of impressive quality, and in some instances, they attain a photorealistic standard. However, to harness these newly generated data for the purpose of training supervised models, it requires the use of labels. These labels play an integral role in identifying the elements contained in an image and are an essential tool in training learning models to recognize various objects. One of the most challenging tasks is generating labels intended for semantic segmentation – deciphering which pixels in an image are associated with specific objects. At deepsense.ai, we have embraced the challenge of devising a novel approach that simultaneously generates images complete with precise segmentation masks. We are sharing the results of our work below.

Demanding data for semantic segmentation

Obtaining segmentation masks is a challenging process. Even though advanced tools for automatic labeling exist, such as Label Studio or CVAT, they have certain limitations.
Recently Meta AI was in the spotlight with the introduction of a groundbreaking new model, the Segment Anything Model (SAM) [1], along with the SA-1B Dataset consisting of 11m images with 1.1bn mask annotations. Despite the availability of this extensive dataset, there remains a persistent need for customized datasets tailored to specific, often niche tasks.

The ability to generate large, annotated datasets is critical in effectively training deep-learning models for semantic segmentation. Objects within images can have a wide range of appearances, such as varying shapes, sizes, and colors. This variability can make it difficult for automated segmentation algorithms to consistently identify and segment objects, especially when objects have overlapping or ambiguous boundaries. We often deal with high-quality images depicting many objects of diverse classes. Thus, it is still common to label them manually. Unfortunately, this is time-consuming, and therefore expensive.

Data generation before diffusion models

Many researchers and Data Science practitioners have faced up to the challenge of overcoming the above-mentioned limitations. Their methods take advantage of the power of deep neural networks and GANs to generate high-quality images with accurate segmentation masks.

Generative Adversarial Networks (GANs) have shown impressive results in generating high-quality images that resemble real-world images. However, generating images with accurate segmentation masks has remained a challenge. Traditional methods for generating segmented images require separate models for segmentation and image generation, leading to slower and less efficient processes.

In “DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort”, Zhang et al. [2] demonstrated how the latent code of GANs can be deciphered to produce a semantic segmentation of the image. When trained to generate images, GANs must acquire a sophisticated understanding of the semantics of objects in order to produce diverse and lifelike examples. Researchers used a pre-trained GAN to generate a few images, which were then labeled by a human annotator. The next step was to train a simple ensemble of MLP classifiers on the feature vectors of the GAN to interpret the knowledge in the pixel feature vector into the pixel label.

To improve the accuracy and efficiency of semantic segmentation, researchers are investigating novel approaches that leverage recent advances in deep learning. One of the areas of focus for researchers is the integration of multimodal inputs, such as using both RGB images and text prompt, to improve the creation of segmentation masks. By combining these inputs, researchers hope to capture more comprehensive information about the scene.

One of the most interesting proposals modifies the CLIP model. CLIPSeg [5], proposed by Timo Lueddecke and Alexander Ecker, can generate image segmentations based on arbitrary prompts at test time, including text or image prompts. The system extended the CLIP to include a transformer-based decoder for dense prediction. It enables a unified model for three common segmentation tasks: referring expression segmentation, zero-shot segmentation, and one-shot segmentation. The system was trained on an extended version of the PhraseCut dataset. It is capable of generating a binary segmentation map for an image based on a text prompt or an additional image expressing the query.

Generation of images with segmentation masks using diffusion models

More recent approaches to data and annotation creations touch on the topic of Stable Diffusion or simply parts of its architecture. We have covered the exciting subject of diffusion models and how to use them in our comprehensive earlier post, and we highly recommend checking it out here [3] and [4] to dive deeper into this topic.

One such idea worth mentioning is the DiffuMask [6] technique, which utilizes attention maps generated by the diffusion model to create highly realistic images with corresponding mask annotations. It employs cross-attention information guided by text to pinpoint regions specific to certain classes or words.

At deepsense.ai, we took a different approach to exploiting the incredible capabilities of diffusion models to generate data. Our approach involves integrating an extra component into the Stable Diffusion architecture to generate segmentation masks alongside images during the creation process. This additional component operates on the hidden states of the Stable Diffusion autoencoder, producing the corresponding segmentation mask. Notably, the extra component can be trained independently of the original autoencoder, leading to more efficient training. Additionally, the model may be lightweight as it leverages some of the weights from the original decoder.

The visual description of the proposed approach is shown in the above figure. The crucial parts of the Stable Diffusion are not changed. The segmentation component is integrated with the VAE decoder.

The training procedure of our approach is shown in Fig. 5. Our training process involves a pair of input data: an image and its corresponding segmentation mask. The image is encoded into a latent space and then decoded by the Stable Diffusion autoencoder. The hidden states of the decoder are then passed through the Segmentator module to generate the segmentation mask. The goal of the training is to teach the model to generate output masks that are similar to the input masks. During the training, only the Segmentator module is trained, and the weights of the Stable Diffusion autoencoder are frozen.

By using this approach, we are able to efficiently train the segmentation module while leveraging the power of the pre-trained autoencoder. This technique significantly reduces training time and computational complexity in comparison to the training of the external segmentation model.

Pet dataset

We started with simple photos of pets from the Oxford-IIIT Pet Dataset [8]. This dataset contains several thousand images of cats and dogs with corresponding segmentation masks. We trained our model to accurately reproduce the segmentation masks of pets. Following the completion of the training process, we were able to successfully generate pairs of images and corresponding segmentation masks. The results below demonstrate the ability of our model to identify the pets in the image:

While it performs well for simple examples, it remains to be seen how well it fares in more complex scenarios. Let’s explore its effectiveness in more intriguing cases.

Pink dog with green dots	Cat under water	Dog on a tank	Cat flying in deep space

Cityscapes dataset

While generating pets may be a relatively straightforward task with limited practical applications, our approach can also be extended to much more complex scenarios. Let’s investigate its effectiveness in the context of urban scenes.

We focus on the Cityscapes dataset [9], which is widely used in computer vision tasks. The dataset contains several thousand images from a few cities in Germany. Each image is annotated with pixel-level semantic labels for 30 different object categories. The images from this dataset have unique characteristics, such as lightning or the layout, which rule out the possibility of completing data from other urban datasets. But by using the presented approach, we can generate any number of annotated data exactly in the style of Cityscapes.

To do this, we need to train not only the segmentation module but also retrain Stable Diffusion to be able to generate pseudo-Cityscapes images. We accomplish this by training the LoRA model described in our previous blog post [3].

In the table below, we present the results for newly generated images with corresponding masks:

Combining the different diffusion model methods, we are able to generate the annotated data in exactly the style we want!

Conclusions

In this post, we discussed different approaches to generating images along with segmentation masks. We proposed a novel method, which changes the original Stable Diffusion architecture by adding a Segmentator module. This custom approach enables us to reduce the training time needed in comparison to the external segmentation model with similar outputs.

References:

[1] Segment Anything, Meta AI Research 
[2] “DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort” Zhang et. al. 
[3] “Diffusion models in practice. Part 1: The tools of the trade”
[4] “Data generation with diffusion models – part 1”
[5] “Image Segmentation Using Text and Image Prompts” Lüddecke, Ecker
[6] ”DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic
Segmentation Using Diffusion Models”
[7] https://deepsense.ai/wp-content/uploads/2023/07/2112.10752.pdf
[8]: https://www.robots.ox.ac.uk/~vgg/data/pets/
[9]: https://www.cityscapes-dataset.com/dataset-overview/

When large language models come into play as part of building a competitive advantage within services or products, many additional questions arise. Effective implementation that brings real business value requires the analysis of aspects such as reliability, high availability, security, data privacy and many more. Such research is time-consuming, which is why deepsense.ai comes with some of it done for you already.

For the purposes of our projects, we have reviewed the key aspects related to the use of OpenAI models. In this article, we share our insights related to two main ways of accessing the OpenAI models, both directly from the organization’s API and via Microsoft Azure OpenAI Service. We review the two options for the central block of this kind of platform that can be covered by the services offered by the two.

What is OpenAI and what is the role of Microsoft in the equation?

OpenAI is an artificial intelligence company founded in December 2015. Its influence in terms of making AI available to a large community of users and becoming part of pop culture has been nothing short of spectacular. Each series of their famous GPT models quickly became popular, giving rise to excitement, doubts, concerns, and… new business opportunities!

The partnership between OpenAI and Microsoft dates back to November 2021, and it seems that their common goal and a shared ambition to responsibly advance cutting-edge AI research and democratize AI as a new technology platform only strengthens this collaboration. Microsoft is one of the biggest public cloud providers. Their immense cloud resources and robust services have supported OpenAI research in their collaboration, but are not limited only to that company. Currently Microsoft offers resources to other Azure users, allowing them to benefit from the results of the cooperation with OpenAI within a well-known, mature and safe cloud environment.

OpenAI vs Azure OpenAI Service: The two options in detail

OpenAI has released a simple and intuitive API to interact with their models. The requests are handled by OpenAI servers. Such an approach allows us to focus the team’s efforts on research rather than infrastructure and computing challenges. The models were developed as a text in, text out interface based on text prompts. The pattern provided in the request decides. The API and Python SDK openai package can be used.

As far as data security is concerned, the situation changed in April 2023. OpenAI ChatGPT had received heavy criticism from both users and experts concerning its further data usage and retention policies. Currently it is easier to ensure data are not being used to train the models by default. The OpenAI blog confirms that, having chosen to disable history for ChatGPT, new conversations are going to be retained for 30 days before permanent deletion only for monitoring abuse. The TLS communication of OpenAI API allows for in-transit encryption for customer-to-OpenAI requests and responses.

From the perspective of Microsoft Azure, having the early feedback lessons from people who are pushing the envelope and are on the cutting edge of this gives us a great deal of insight into and a head start on what is going to be needed as this infrastructure moves forward. Microsoft currently provides exciting offers that combine the main advantages of the two parties via Microsoft Azure OpenAI Service.

As we can read in the article by Eric Boyd, Corporate Vice President, AI Platform: The power of large-scale generative AI models with the enterprise promises customers have come to expect from our Azure cloud. The service enables the customers to run the same models as OpenAI, benefiting from the enterprise-scale features of the Azure cloud.

In terms of data security, apart from private networking and regional availability, Azure provides privacy and confidentiality when one entrusts them with data. Azure fulfills the following expectations:

• Your data is not shared without your consent and is not mined by Microsoft Azure.
• The data is processed only with the owner’s consent, and this regulation also applies to Microsoft-authorized subcontractors or subprocessors. The constraints and contractual privacy commitments are sustained.
• The data is removed from the cloud once you leave the Azure service or your subscription expires.
• The data is secured with state-of-the-art encryption mechanisms both at rest and in transit.

To sum up, if you need to ensure no further usage of your data, Microsoft, as a provider, creates a safe place for your data. If you already possess any data on Azure servers, the choice of service is low-hanging fruit. Additionally, their safety will not be affected in Azure OpenAI Service. As this is a very explicit declaration about safety, we recommend using this service especially when data vulnerability is of concern.

OpenAI models from both providers

There are several engines (corresponding to the models) as their capabilities (and customization opportunities through fine-tuning) differ. For both OpenAI API and Azure, there are a few categories of models which are ready-to-use or allow zero- or few-shot learning depending on the use case. One can always choose between alternative categories:

• basic language models that are adaptable to content generation, summarization, and natural language to code translation.
• the costs of few-shot learning are reduced – pre-trained language models that can be further trained on specific tasks or domains. This ultimately saves money and improves performance through shorter prompts (no need to put examples in the prompt – decreases token usage).
• embedding model series – their responsibility is to convert the text into vector representation to make it possible to measure the relatedness of text strings and be used for further tasks such as search, classification, clustering, diversity management, recommendations and anomaly detection.
• fine-tuned version (with text or code prefix for text and code generation respectively).

The models released by OpenAI have been organized into series corresponding to experiments and the respective tasks the engines perform and the engine categories listed above partially overlap with the series but represent a different quality.

Although one can find the entire list of models accessible via OpenAI API in their docs, we are going to limit our attention to the models that are also available in the Azure OpenAI service. Shortly, we will skip the Dall-E, Whisper, Moderation and some GPT-3.5 series models – they are not explicitly accessible in the latter service and are beyond the scope of this related post summarizing aspects of LLM API usage. We compare the context length and time range of the training data set (the maximum number of tokens if approximated to thousands). The details of the models are far beyond the scope of this post, yet we want to provide the basic intuition behind the engines that can be used.

The GPT-3 base models available are Ada, Babbage, Curie and Davinci – the ordering is not accidental and represents the capability vs speed trade-off with Ada as the fastest engine and Davinci as the most powerful. All models were trained on data from up to October 2019 and have a maximum of 2k tokens.

Even though all GPT-3 based engines are usually substituted with the more powerful GPT-3.5 series model, they are also served by fine-tuned alternatives (text prefix) and are available for embedding tasks or are fine-tunable with user data.

For Azure OpenAI Service, these are the models available for getting embeddings. The same holds true for OpenAI, but they also recommend a second generation embedding model, text-embedding-ada-002 (8k tokens) designed to replace the previous generation of embedding models at significantly lower cost.

If one is interested in a modern series of language models optimized for chat but suitable for traditional completion tasks as well, there are at least two served by both providers. The GPT-3.5 series has a flagship model gpt-3.5-turbo (4k tokens) – it is recommended as a cost-effective and the most capable model. Please take into account the fact that, though OpenAI API supports multiple countries (without explicitly mentioning the server regions) not all regions of Azure OpenAI Service enable this engine (for example Central US, in contrast to Eastern US or Western Europe).

Another alternative is GPT-4 of an even higher series that has recently aroused the interest of the public. At the time of writing, it is available in the API documentation as limited beta. In the case of this engine, the regional support in Azure is narrowed down to the Eastern and South Central regions in the case of the US and is not available in other regions. One should consider the fact that currently there is a waitlist to join to get access to this engine API in Open API. This engine is a large multimodal model with 8k and 32k maximum token variants and is meant to solve difficult problems and perform with advanced reasoning capabilities and provide results at a greater level of accuracy compared to the previous models thanks to its broad general knowledge.

Code generation (Codex)

The situation with LLMs understanding and generating code changed in March 2023. For users who are interested in using the engines that were commonly used, namely the code-davinci and code-cushman models (8k and 2k tokens respectively and the speed advantage of the latter), they are still available in Azure OpenAI Service. The same engines are deprecated in OpenAI API and the chat models are recommended for the task – the capabilities remain similar..

Having discussed the details of the various models above, we present a table to summarize the high-level specifications of the models.

Series	Latest model	Max tokens (by 1.024)	Training data
GPT-3	davinci	2k	Up to Oct 2019
	curie
	babbage
	ada
	code-cushman		N/A
	code-davinci-001	8k	Up to Jun 2021
GPT-3.5	text-davinci-002	4k
	gpt-3.5-turbo(-*)	4k	Up to Sep 2021
GPT-4	gpt-4(-*)	8k, 32k	Up to Sep 2021

Table 1. OpenAI model sizes and training data overview. Source: OpenAI docs.

OpenAI vs Microsoft Azure OpenAI Service: similarities 

At this point in our analysis, it is worth mentioning two main similarities.

API Compatibility

Co-development of the APIs with OpenAI and ensuring compatibility to make the transition and integration of the models seamless is the strategy behind Microsoft Azure OpenAI Service. Thus, switching between both services should be fairly easy and might simply involve an alternative configuration to your codebase. If the business use-case and cost optimization strategy is suitable, dynamic migration between providers comes into play.

Limitations and safety

OpenAI claims that the models may encode social biases (negative sentiment towards certain groups/stereotypes). These issues are addressable for each provider but in a different manner (please see the section about differences). Moreover, the models lack knowledge of events which took place after August 2020.

OpenAI vs Microsoft Azure OpenAI Service: differences

In addition to the characteristics of both solutions already described in this article, it is worth noting three basic differences.

Pricing

Pricing is one of the key differences between the providers. The Microsoft Azure calculator helps to estimate pricing of fine-tuning, as well as hosting a fine-tuned model, whereas with OpenAI API one is charged only for the tokens used in requests to that model. The pricing of single API requests (per 1k tokens) of both providers is alike..

Regional availability

Based on the OpenAI API usage policies, all customer data is processed and stored in the US. If the volume of your data is large enough, it is recommended to keeping the compute power close to the data in order to avoid moving it excessively is recommended, so as to avoid network transit overhead. This seems to suggest that the choice of Azure OpenAI Service would be best if you need elasticity or a particular region of availability.

Limitations and safety

From the perspective of an Azure OpenAI Service user, the recommended text-embedding-ada-002 model is available in Azure OpenAI, but the number of tokens used might be limited (see this thread).

The model used with OpenAI API can be supported with their Moderation model to classify and prevent content that is hateful, harmful, violent, sexual or discriminates against minorities. Put simply, it must be compliant with the usage policies of OpenAI. Similarly, the content requirements must be met in the case of its Azure counterpart, so the content policies are intended to improve the safety of the platform for both the input and output of the engines, and are always explicitly filtered.

For more information, see and compare the OpenAI safety standards and the Azure service Responsible AI section in the docs.

Open AI models: final thoughts

A multitude of available models allow for various implementations for your services or products and provide a dynamic approach to scaling and optimizing costs and performance. The adoption of OpenAI solutions creates exciting new business opportunities for various industries. On the other hand, there is an exciting world of alternatives, and our experts can guide you on your journey. Let us know if our AI development agency can help build your vision incorporating large language models!