Smart Data Warehouse Limited

Microsoft Fabric Has Made Azure Data Factory More Relevant Than Ever — And Skilled Data Engineers Are Back in High Demand

With the rise of Microsoft Fabric, Azure Data Factory (ADF) has once again become one of the most important and widely used tools in modern data engineering.

As a result, data engineers who truly understand how to use ADF effectively are in huge demand again.

This week in our Data Engineering Masterclass, we revisited one of the practical areas where experienced engineers often struggle when running operational pipelines efficiently:

choosing the right approach for processing multiple files inside a folder.
In this short note, we highlight when each approach should be used, and we’ve also shared some of the insights demonstrated during our class session.

When a Folder Contains Multiple Files, Efficiency Matters
Moving files one by one is rarely the best option.

Two common approaches are:

1. Using a JSON Lookup File

Best used when: The files arriving in ADLS retain the same names
You want a simple, controlled lookup mechanism inside your pipeline

2. Using the GetMetadata Activity

Best used when: Files may arrive with different or unpredictable names
You need the pipeline to dynamically detect and process whatever is present
Both approaches look simple on the surface, but sound knowledge of when and how to apply each one is essential for building efficient, production‑grade pipelines.

These are exactly the kinds of practical, real‑world skills we teach in our program.

New Cohort Starts in May

Registration for our next batch is already open.

If you’re interested in joining the class, feel free to reach out.

🔹 Why Your Fact Table Deserves Better ETL Design

One thing many engineers still struggle with is keeping the fact table clean, accurate, and aligned with only the active surrogate keys from the dimension table.

This is the heart of every analytical workload — and when it’s wrong, everything downstream suffers.

The tricky part is that many overcomplicate this process. But with ADF, you can still implement SCD Type 2 cleanly, pass the correct surrogate keys into your fact table, and maintain a simple, scalable pipeline that anyone can understand.

We’ll be breaking this down at the end of this month in our ADF & Fabric training — simplifying the exact areas where most people get stuck.

If you’ve been wanting to master this and build pipelines that truly shine in production, now is the time to register.

Forecasting with Returns vs Differencing — Which Is Easier to Reverse?

Today, our data science students wrapped up their time series forecasting presentations — a proud moment for Smart Data Warehouse Limited.

As we prepare to dive into multivariate regression next week, I took time to connect with students who preferred the shortcut method of differencing to stationarize their data.

Their reason?

Simplicity. Reversing forecasted values back to real prices felt more intuitive with differencing than with returns, which require compounding and scaling.

That concern sparked an extra hour of hands-on re-demonstration — walking through how to convert forecasted returns back into actual price levels.

Special thanks to our senior analysts Henry Nwachukwu and Chioma Oluigbo GMNSE for guiding the students through this nuanced topic with clarity and patience.

For friendly trainings and data consultations, reach out to Smart Data Warehouse Limited at [email protected].