This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some of the topics that you will find here.
…
continue reading
The podcast about Python and the people who make it great
…
continue reading
![Artwork](/static/images/128pixel.png)
1
The Evolution of DataOps: Insights from DataKitchen's CEO
53:30
53:30
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
53:30
Summary In this episode of the Data Engineering Podcast, host Tobias Macey welcomes back Chris Berg, CEO of DataKitchen, to discuss his ongoing mission to simplify the lives of data engineers. Chris explains the challenges faced by data engineers, such as constant system failures, the need for rapid changes, and high customer demands. Chris delves …
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Achieving Data Reliability: The Role of Data Contracts in Modern Data Management
49:26
49:26
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
49:26
Summary Data contracts are both an enforcement mechanism for data quality, and a promise to downstream consumers. In this episode Tom Baeyens returns to discuss the purpose and scope of data contracts, emphasizing their importance in achieving reliable analytical data and preventing issues before they arise. He explains how data contracts can be us…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
How Generative AI Is Impacting Data Engineering Teams
54:45
54:45
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
54:45
Summary Generative AI has rapidly gained adoption for numerous use cases. To support those applications, organizational data platforms need to add new features and data teams have increased responsibility. In this episode Lior Gavish, co-founder of Monte Carlo, discusses the various ways that data teams are evolving to support AI powered features a…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
The Role of Product Managers in Data-Centric Organizations
52:58
52:58
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
52:58
Summary In this episode Praveen Gujar, Director of Product at LinkedIn, talks about the intricacies of product management for data and analytical platforms. Praveen shares his journey from Amazon to Twitter and now LinkedIn, highlighting his extensive experience in building data products and platforms, digital advertising, AI, and cloud services. H…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Neon: A Serverless And Developer Friendly Postgres
57:43
57:43
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
57:43
Summary Postgres is one of the most widely respected and liked database engines ever. To make it even easier to use for developers to use, Nikita Shamgunov decided to makee it serverless, so that it can scale from zero to infinity. In this episode he explains the engineering involved to make that possible, as well as the numerous details that he an…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Improve Data Quality Through Engineering Rigor And Business Engagement With Synq
59:48
59:48
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
59:48
Summary This episode features an insightful conversation with Petr Janda, the CEO and founder of Synq. Petr shares his journey from being an engineer to founding Synq, emphasizing the importance of treating data systems with the same rigor as engineering systems. He discusses the challenges and solutions in data reliability, including the need for …
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Stitching Together Enterprise Analytics With Microsoft Fabric
53:23
53:23
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
53:23
Summary Data lakehouse architectures have been gaining significant adoption. To accelerate adoption in the enterprise Microsoft has created the Fabric platform, based on their OneLake architecture. In this episode Dipti Borkar shares her experiences working on the product team at Fabric and explains the various use cases for the Fabric service. Ann…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Being Data Driven At Stripe With Trino And Iceberg
53:20
53:20
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
53:20
Summary Stripe is a company that relies on data to power their products and business. To support that functionality they have invested in Trino and Iceberg for their analytical workloads. In this episode Kevin Liu shares some of the interesting features that they have built by combining those technologies, as well as the challenges that they face i…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
X-Ray Vision For Your Flink Stream Processing With Datorios
42:22
42:22
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
42:22
Summary Streaming data processing enables new categories of data products and analytics. Unfortunately, reasoning about stream processing engines is complex and lacks sufficient tooling. To address this shortcoming Datorios created an observability platform for Flink that brings visibility to the internals of this popular stream processing system. …
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Practical First Steps In Data Governance For Long Term Success
1:00:41
1:00:41
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
1:00:41
Summary Modern businesses aspire to be data driven, and technologists enjoy working through the challenge of building data systems to support that goal. Data governance is the binding force between these two parts of the organization. Nicola Askham found her way into data governance by accident, and stayed because of the benefit that she was able t…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Data Migration Strategies For Large Scale Systems
1:00:00
1:00:00
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
1:00:00
Summary Any software system that survives long enough will require some form of migration or evolution. When that system is responsible for the data layer the process becomes more challenging. Sriram Panyam has been involved in several projects that required migration of large volumes of data in high traffic environments. In this episode he shares …
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Zenlytic Is Building You A Better Coworker With AI Agents
54:19
54:19
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
54:19
Summary The purpose of business intelligence systems is to allow anyone in the business to access and decode data to help them make informed decisions. Unfortunately this often turns into an exercise in frustration for everyone involved due to complex workflows and hard-to-understand dashboards. The team at Zenlytic have leaned on the promise of la…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Release Management For Data Platform Services And Logic
20:09
20:09
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
20:09
Summary Building a data platform is a substrantial engineering endeavor. Once it is running, the next challenge is figuring out how to address release management for all of the different component parts. The services and systems need to be kept up to date, but so does the code that controls their behavior. In this episode your host Tobias Macey ref…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Barking Up The Wrong GPTree: Building Better AI With A Cognitive Approach
54:17
54:17
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
54:17
Summary Artificial intelligence has dominated the headlines for several months due to the successes of large language models. This has prompted numerous debates about the possibility of, and timeline for, artificial general intelligence (AGI). Peter Voss has dedicated decades of his life to the pursuit of truly intelligent software through the appr…
…
continue reading
Summary Generative AI promises to accelerate the productivity of human collaborators. Currently the primary way of working with these tools is through a conversational prompt, which is often cumbersome and unwieldy. In order to simplify the integration of AI capabilities into developer workflows Tsavo Knott helped create Pieces, a powerful collecti…
…
continue reading
Summary Generative AI has rapidly transformed everything in the technology sector. When Andrew Lee started work on Shortwave he was focused on making email more productive. When AI started gaining adoption he realized that he had even more potential for a transformative experience. In this episode he shares the technical challenges that he and his …
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Designing A Non-Relational Database Engine
1:16:02
1:16:02
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
1:16:02
Summary Databases come in a variety of formats for different use cases. The default association with the term "database" is relational engines, but non-relational engines are also used quite widely. In this episode Oren Eini, CEO and creator of RavenDB, explores the nuances of relational vs. non-relational engines, and the strategies for designing …
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Establish A Single Source Of Truth For Your Data Consumers With A Semantic Layer
56:23
56:23
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
56:23
Summary Maintaining a single source of truth for your data is the biggest challenge in data engineering. Different roles and tasks in the business need their own ways to access and analyze the data in the organization. In order to enable this use case, while maintaining a single point of access, the semantic layer has evolved as a technological sol…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Adding Anomaly Detection And Observability To Your dbt Projects Is Elementary
50:44
50:44
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
50:44
Summary Working with data is a complicated process, with numerous chances for something to go wrong. Identifying and accounting for those errors is a critical piece of building trust in the organization that your data is accurate and up to date. While there are numerous products available to provide that visibility, they all have different technolo…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+
55:40
55:40
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
55:40
Summary A core differentiator of Dagster in the ecosystem of data orchestration is their focus on software defined assets as a means of building declarative workflows. With their launch of Dagster+ as the redesigned commercial companion to the open source project they are investing in that capability with a suite of new features. In this episode Pe…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Reconciling The Data In Your Databases With Datafold
58:14
58:14
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
58:14
Summary A significant portion of data workflows involve storing and processing information in database engines. Validating that the information is stored and processed correctly can be complex and time-consuming, especially when the source and destination speak different dialects of SQL. In this episode Gleb Mezhanskiy, founder and CEO of Datafold,…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Version Your Data Lakehouse Like Your Software With Nessie
40:55
40:55
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
40:55
Summary Data lakehouse architectures are gaining popularity due to the flexibility and cost effectiveness that they offer. The link that bridges the gap between data lake and warehouse capabilities is the catalog. The primary purpose of the catalog is to inform the query engine of what data exists and where, but the Nessie project aims to go beyond…
…
continue reading
Summary Artificial intelligence technologies promise to revolutionize business and produce new sources of value. In order to make those promises a reality there is a substantial amount of strategy and investment required. Colleen Tartow has worked across all stages of the data lifecycle, and in this episode she shares her hard-earned wisdom about h…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Find Out About The Technology Behind The Latest PFAD In Analytical Database Development
56:01
56:01
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
56:01
Summary Building a database engine requires a substantial amount of engineering effort and time investment. Over the decades of research and development into building these software systems there are a number of common components that are shared across implementations. When Paul Dix decided to re-write the InfluxDB engine he found the Apache Arrow …
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Using Trino And Iceberg As The Foundation Of Your Data Lakehouse
58:46
58:46
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
58:46
Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Multiple open source projects and vendors have been working together to make this vision a reality. In this episode Dain Sundstrom, CTO of Starburst, explains how the combinatio…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Data Sharing Across Business And Platform Boundaries
59:56
59:56
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
59:56
Summary Sharing data is a simple concept, but complicated to implement well. There are numerous business rules and regulatory concerns that need to be applied. There are also numerous technical considerations to be made, particularly if the producer and consumer of the data aren't using the same platforms. In this episode Andrew Jefferson explains …
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Tackling Real Time Streaming Data With SQL Using RisingWave
56:55
56:55
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
56:55
Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer. In this episode Yingjun Wu explains how it is architected to power analytical workflows on continu…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Build A Data Lake For Your Security Logs With Scanner
1:02:38
1:02:38
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
1:02:38
Summary Monitoring and auditing IT systems for security events requires the ability to quickly analyze massive volumes of unstructured log data. The majority of products that are available either require too much effort to structure the logs, or aren't fast enough for interactive use cases. Cliff Crosland co-founded Scanner to provide fast querying…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Modern Customer Data Platform Principles
1:01:33
1:01:33
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
1:01:33
Summary Databases and analytics architectures have gone through several generational shifts. A substantial amount of the data that is being managed in these systems is related to customers and their interactions with an organization. In this episode Tasso Argyros, CEO of ActionIQ, gives a summary of the major epochs in database technologies and how…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Pushing The Limits Of Scalability And User Experience For Data Processing WIth Jignesh Patel
50:26
50:26
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
50:26
Summary Data processing technologies have dramatically improved in their sophistication and raw throughput. Unfortunately, the volumes of data that are being generated continue to double, requiring further advancements in the platform capabilities to keep up. As the sophistication increases, so does the complexity, leading to challenges for user ex…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Designing Data Platforms For Fintech Companies
47:57
47:57
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
47:57
Summary Working with financial data requires a high degree of rigor due to the numerous regulations and the risks involved in security breaches. In this episode Andrey Korchack, CTO of fintech startup Monite, discusses the complexities of designing and implementing a data platform in that sector. Announcements Hello and welcome to the Data Engineer…
…
continue reading
Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Operating it at scale, however, is notoriously challenging. Elad Eldor has experienced these challenges first-hand, leading to his work writing the book "Kafka: : Troubleshooting in Production". In this episode he hi…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Adding An Easy Mode For The Modern Data Stack With 5X
56:12
56:12
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
56:12
Summary The "modern data stack" promised a scalable, composable data platform that gave everyone the flexibility to use the best tools for every job. The reality was that it left data teams in the position of spending all of their engineering effort on integrating systems that weren't designed with compatible user experiences. The team at 5X unders…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Run Your Own Anomaly Detection For Your Critical Business Metrics With Anomstack
51:18
51:18
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
51:18
Summary If your business metrics looked weird tomorrow, would you know about it first? Anomaly detection is focused on identifying those outliers for you, so that you are the first to know when a business critical dashboard isn't right. Unfortunately, it can often be complex or expensive to incorporate anomaly detection into your data platform. And…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Designing Data Transfer Systems That Scale
1:03:57
1:03:57
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
1:03:57
Summary The first step of data pipelines is to move the data to a place where you can process and prepare it for its eventual purpose. Data transfer systems are a critical component of data enablement, and building them to support large volumes of information is a complex endeavor. Andrei Tserakhau has dedicated his careeer to this problem, and in …
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Addressing The Challenges Of Component Integration In Data Platform Architectures
29:43
29:43
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
29:43
Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. One of the core complexities that needs to be addressed is the fractal set of integrations that need to be managed across the individual components. In this episode Tobias Macey shares his thoughts on the challenges that he is faci…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Unlocking Your dbt Projects With Practical Advice For Practitioners
1:16:04
1:16:04
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
1:16:04
Summary The dbt project has become overwhelmingly popular across analytics and data engineering teams. While it is easy to adopt, there are many potential pitfalls. Dustin Dorsey and Cameron Cyr co-authored a practical guide to building your dbt project. In this episode they share their hard-won wisdom about how to build and scale your dbt projects…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Enhancing The Abilities Of Software Engineers With Generative AI At Tabnine
1:07:52
1:07:52
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
1:07:52
Summary Software development involves an interesting balance of creativity and repetition of patterns. Generative AI has accelerated the ability of developer tools to provide useful suggestions that speed up the work of engineers. Tabnine is one of the main platforms offering an AI powered assistant for software engineers. In this episode Eran Yaha…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Shining Some Light In The Black Box Of PostgreSQL Performance
54:52
54:52
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
54:52
Summary Databases are the core of most applications, but they are often treated as inscrutable black boxes. When an application is slow, there is a good probability that the database needs some attention. In this episode Lukas Fittl shares some hard-won wisdom about the causes and solution of many performance bottlenecks and the work that he is doi…
…
continue reading
Summary Databases are the core of most applications, whether transactional or analytical. In recent years the selection of database products has exploded, making the critical decision of which engine(s) to use even more difficult. In this episode Tanya Bragin shares her experiences as a product manager for two major vendors and the lessons that she…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Defining A Strategy For Your Data Products
1:03:50
1:03:50
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
1:03:50
Summary The primary application of data has moved beyond analytics. With the broader audience comes the need to present data in a more approachable format. This has led to the broad adoption of data products being the delivery mechanism for information. In this episode Ranjith Raghunath shares his thoughts on how to build a strategy for the develop…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Reducing The Barrier To Entry For Building Stream Processing Applications With Decodable
1:08:29
1:08:29
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
1:08:29
Summary Building streaming applications has gotten substantially easier over the past several years. Despite this, it is still operationally challenging to deploy and maintain your own stream processing infrastructure. Decodable was built with a mission of eliminating all of the painful aspects of developing and deploying stream processing systems …
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Using Data To Illuminate The Intentionally Opaque Insurance Industry
51:58
51:58
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
51:58
Summary The insurance industry is notoriously opaque and hard to navigate. Max Cho found that fact frustrating enough that he decided to build a business of making policy selection more navigable. In this episode he shares his journey of data collection and analysis and the challenges of automating an intentionally manual industry. Announcements He…
…
continue reading
Summary Artificial intelligence applications require substantial high quality data, which is provided through ETL pipelines. Now that AI has reached the level of sophistication seen in the various generative models it is being used to build new ETL workflows. In this episode Jay Mishra shares his experiences and insights building ETL pipelines with…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Powering Vector Search With Real Time And Incremental Vector Indexes
59:16
59:16
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
59:16
Summary The rapid growth of machine learning, especially large language models, have led to a commensurate growth in the need to store and compare vectors. In this episode Louis Brandy discusses the applications for vector search capabilities both in and outside of AI, as well as the challenges of maintaining real-time indexes of vector data. Annou…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Building Linked Data Products With JSON-LD
1:01:31
1:01:31
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
1:01:31
Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. Linked data technologies provide a means of tightly coupling metadata with raw information. In this episode Brian Platz explains how JSON-LD can be used as a shared representation of linked data for buildi…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
An Overview Of The State Of Data Orchestration In An Increasingly Complex Data Ecosystem
1:01:26
1:01:26
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
1:01:26
Summary Data systems are inherently complex and often require integration of multiple technologies. Orchestrators are centralized utilities that control the execution and sequencing of interdependent operations. This offers a single location for managing visibility and error handling so that data platform engineers can manage complexity. In this ep…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Eliminate The Overhead In Your Data Integration With The Open Source dlt Library
42:13
42:13
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
42:13
Summary Cloud data warehouses and the introduction of the ELT paradigm has led to the creation of multiple options for flexible data integration, with a roughly equal distribution of commercial and open source options. The challenge is that most of those options are complex to operate and exist in their own silo. The dlt project was created to elim…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Building An Internal Database As A Service Platform At Cloudflare
1:01:10
1:01:10
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
1:01:10
Summary Data persistence is one of the most challenging aspects of computer systems. In the era of the cloud most developers rely on hosted services to manage their databases, but what if you are a cloud service? In this episode Vignesh Ravichandran explains how his team at Cloudflare provides PostgreSQL as a service to their developers for low lat…
…
continue reading
![Artwork](/static/images/128pixel.png)
1
Harnessing Generative AI For Creating Educational Content With Illumidesk
54:52
54:52
「あとで再生する」
「あとで再生する」
リスト
気に入り
気に入った
54:52
Summary Generative AI has unlocked a massive opportunity for content creation. There is also an unfulfilled need for experts to be able to share their knowledge and build communities. Illumidesk was built to take advantage of this intersection. In this episode Greg Werner explains how they are using generative AI as an assistive tool for creating e…
…
continue reading