Is It Time to Say Goodbye to Data Engineers? - The Data Engineering Dilemma

Ever since tools like SSIS came onto the scene, vendors and business leaders have been on a mission to remove what they see as the biggest roadblock to data-driven decision-making: data engineers.

Or their counterparts—DBAs, ETL Developers, and Data Architects.

Sure, not everyone says it so explicitly, but you can see it in vendor marketing and in the decisions made by the business.

I remember talking to a veteran data expert who’s been in the field for three decades. They told me that when SSIS first launched, people were genuinely afraid for their jobs. The idea that you could just drag-and-drop tasks that once required code was nerve-racking. But if you’ve used SSIS, well, you know the truth.

To some extent, I get why the idea is appealing. When a leader requests a report, a software engineer wants to modify an application table, or a data scientist wants to explore a new dataset, who’s the one slowing down the project?

The data engineers.

With their rules. Their governance. Their insistence on building robust, scalable data pipelines instead of quick fixes.

They really slow down the workflow!

If you enjoyed this video, check out some of my other top videos.

Data Modeling - Walking Through How To Data Model As A Data Engineer - Dimensional Modeling 101
   • Data Modeling - Walking Through How T...  

Databases Vs Data Warehouses Vs Data Lakes - What Is The Difference And Why Should You Care?
   • Databases Vs Data Warehouses Vs Data ...  

If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.

seattledataguy.substack.com/​​

If you're looking for a tool to make ingesting data into Snowflake, BigQuery, Iceberg or Databricks easier, then check out Estuary(Full disclosure I am an advisor for them)!
bit.ly/4eQC3oQ



And if you want to support the channel, then you can become a paid member of my newsletter
seattledataguy.substack.com/subscribe


Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio

_____________________________________________________________
Subscribe:    / @seattledataguy  
_____________________________________________________________
About me:
I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consult on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.

*I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you.