"Digitale Wissensbissen": The future of data analysis – A conversation with Christian Schömmer

MashMachine
Podcast
Technology
#DataWarehouse
#DataLake
#DataLakehouse
#TextAI
#RAG
#DigitaleWissensbissenS01E05
Cover Image for "Digitale Wissensbissen": The future of data analysis – A conversation with Christian Schömmer

Introduction

Companies that are able to use their data effectively have a clear competitive advantage. But how can the huge amounts of structured and unstructured data that are generated every day be used sensibly? Especially in connection with generative AI? In our new episode, we invited Christian Schömmer, Senior Partner Manager Data Analytics at AWS, to talk about the challenges and opportunities of modern data architectures. Christian shares his experiences and provides valuable insights into the world of data lakes, data warehouses and the emerging technology of the lakehouse.

Key Points from the episode

  1. On the evolution of data analysis:
    - “The hope that you can now do something with data that is beyond your own ability to analyze it is what drives the whole development.”
    - “In the past, the first use case for enterprise search was simply ‘I'll find stuff’.”
  2. On the challenges and opportunities of Gen AI:
    - “Gen AI has generated a lot of utopian expectations, but there is still a glass ceiling when it comes to reliability.”
    - “The combination of Gen AI and data storage offers a gigantic promise, but also many potential risks.”
  3. About the differences between data warehouse, data lake and lakehouse:
    - “A data warehouse is optimized for structured data and specific queries, while a data lake is more flexible and can accommodate both structured and unstructured data.”
    - “The lakehouse combines the advantages of both worlds and makes it possible to use both structured and unstructured data efficiently.”
  4. About practical implementation and barriers to entry:
    - “The barrier to entry is currently lower than it has been in a long time. There is no reason not to do anything in this area.”
    - “A data scientist would certainly not be a bad start, but the team size has been significantly minimized compared to ten years ago.”

Summary

In our conversation with Christian Schömmer, we delved deep into the world of modern data architectures. A central topic was the evolution of data analysis tools and the role of Gen AI. While Gen AI has raised many utopian expectations, there are still challenges in terms of reliability and integration into existing data structures.

Another focus was on the differences between data warehouses, data lakes and lakehouses. While data warehouses are optimized for structured data and specific queries, data lakes offer a more flexible solution that can accommodate both structured and unstructured data. The lakehouse combines the advantages of both worlds and enables companies to use their data more efficiently.

Christian also emphasized that the barriers to entry for modern data analysis tools are lower than ever. Companies, even small and medium-sized ones, can start using their data in a meaningful way with relatively little effort and without large teams. The availability of out-of-the-box tools and the ability to iterate make it easier to take the first steps towards becoming a data-driven organization.

Overall, the combination of Gen AI and modern data architectures offers a promising future for companies that want to use their data effectively. The risks are present, but the potential benefits far outweigh them. It is an exciting time for anyone involved in data analysis, and the possibilities are almost endless.

(Episode in German only)

MashMachine
MashMachine
AI servant
Artificial intelligence that multiplexes your efforts.

More blog posts

Image

"Digitale Wissensbissen": Generative AI in Business-Critical Processes

After the somewhat critical view of generative AI in the last episode, this time we are looking at the specific application: can generative AI already be integrated into business processes and, if so, how exactly does it work? It turns out that if you follow two or three basic rules, most of the problems fade into the background and the cool possibilities of generative AI can be exploited with relatively little risk. We discuss in detail how we built a compliance application that maximizes the benefits of large language models without sacrificing human control and accountability. (Episode in German)

Image

Bleeding Edge - curse or blessing?

We rely on bleeding edge technologies to drive companies forward with innovative solutions. That's why, in discussions with customers and partners or in our webinars, we are always keen to explain the benefits and possibilities of modern technologies to companies. But we also use AI ourselves: by automating tendering processes, we have been able to save valuable resources and increase efficiency.

Image

Current AI is like a 12-year-old colleague

The technology must be embedded in the process and adapted to the process and not float next to it unconnected. This also applies to large language models - although their "humanity" gives the impression that they can be treated like a colleague rather than a tool. But how many business areas benefit from a colleague with the intellectual capacity of a twelve-year-old?