Most data engineer CVs now carry the same block of tool names: Spark, Airflow, dbt, Kafka, Snowflake or Databricks, and two clouds for good measure. That block tells you very little, because the job underneath it has moved. Managed platforms absorb much of the cluster care that used to fill an engineer’s week, AI assistants produce passable SQL in seconds, and pipelines increasingly feed machine learning features and LLM applications as well as dashboards. The data engineer skills that keep their value are the ones a tool cannot supply, such as modelling data so that “revenue” means the same thing in every report, and noticing when a pipeline is wrong rather than just late.
This guide is for recruiters and hiring managers deciding what to test for. It covers the core technical skills with a practical check for each, the working habits that set good engineers apart, how the skill set has shifted and how to assess it without relying on keywords.
Core technical skills for a data engineer
SQL and analytical data modelling
Good looks like someone who thinks in terms of grain: what one row represents, and what happens when tables of different grain are joined. They can explain fact and dimension tables in plain words and know when a wide denormalised table suits the people querying it better. How to check it: show a join between an orders table and an order-items table that produces inflated totals, and ask what went wrong. Strong candidates spot the fan-out quickly and explain how they would design the tables so it cannot happen.
Python for data work
A data engineer’s Python is less about clever code and more about code that keeps working: calling APIs with pagination and rate limits, retrying sensibly, reading awkward file formats and packaging logic so it can be tested. How to check it: hand over a short ingestion script that pulls from a paginated API and ask what would break first in production. Listen for missed pages, timeouts, credentials in the code and the absence of logging.
Distributed processing with Spark
Spark remains the standard for large transformations, especially on Databricks. Good engineers understand lazy evaluation and why caching helps in one place and wastes memory in another. They also know when Spark is overkill and a warehouse query or DuckDB would do with less to maintain. How to check it: ask them to describe a job where they chose not to use Spark, and why.
Orchestration, dependencies and backfills
Airflow is still the most common orchestrator, with Dagster and Databricks Workflows also in wide use. Scheduling a task is easy. The skill lies in getting dependencies right, making late upstream data visible, and reloading history after a bug fix without double-counting or overloading the source. How to check it: ask how they would reprocess the full history of one table after a logic change while the daily load keeps running. Good answers mention partition-level reloads, keeping the backfill apart from the live schedule and comparing totals before and after.
Testing data code
Software engineers treat untested code as unfinished, and good data engineers now do the same: unit tests for transformation logic, dbt tests on uniqueness and relationships, and checks that run in CI before a change reaches production. How to check it: ask what tests they would write before shipping a new revenue model. People who test by habit answer with specifics straight away.
Streaming and event data
Not every role needs streaming, but where it does, the concepts matter more than the product. Kafka is the usual backbone, often with Spark Structured Streaming or Apache Flink for processing and Debezium for change data capture. A strong engineer understands delivery guarantees, ordering, watermarks and late events. How to check it: ask what happens to a running hourly total when an event arrives after its hour has closed, and how their pipeline would deal with it.
Lakehouse platforms and open table formats
Delta Lake and Apache Iceberg add transactions, schema evolution and time travel to files in object storage. Engineers on Databricks, Snowflake or BigQuery should understand partitioning, file sizes, compaction and clustering, and how storage layout affects speed and cost. How to check it: ask how they would recover a Delta or Iceberg table after a job overwrote it with bad data. Anyone who has worked with these formats will mention time travel or restoring an earlier version.
Governance, access and privacy
Data engineers increasingly own the controls around data as well as the data: role-based access, masking of personal fields, lineage, and catalogues such as Unity Catalog. In India, the Digital Personal Data Protection Act has made it more important to know where personal data sits and who can see it. How to check it: ask how they would give analysts a customer table without exposing phone numbers or email addresses. Look for column masking, secure views or tokenisation rather than “we trust the team”.
Working skills that separate good from average
- Managing upstream relationships. Most broken pipelines start with a change someone else made. Good data engineers get to know the application teams who own source systems, get added to their release notes and agree what counts as a breaking change.
- Telling consumers before they find out. When data is wrong or late, a strong engineer tells the affected teams what is wrong, which reports are affected and when it will be fixed, then writes a short incident note afterwards.
- Knowing what the numbers mean. An engineer who understands that refunds, cancelled orders and test accounts change what “revenue” or “active user” means will build tables analysts can trust. This domain sense often decides whether a pipeline is merely running or actually right.
- Saying no to the wrong request. Data teams are constantly asked for one-off extracts and duplicate tables. Good engineers push back politely and steer people towards shared, documented models, so the platform does not fill up with near-identical tables.
Skills that matter more in 2026, and less
Mattering more. AI coding assistants now write much of the boilerplate SQL and Python, so reviewing generated code critically has become a core skill: catching a join that silently drops rows, or a filter that excludes nulls nobody meant to exclude. Pipelines also feed AI products, which means handling documents and support tickets, chunking text and keeping embeddings in a vector store in step with their source. Open table formats, data contracts and privacy controls have moved from nice to have towards expected as more teams share one platform. Cost awareness matters more on usage-billed platforms.
Mattering less. Running Hadoop clusters, writing MapReduce jobs and hand-managing Hive metastores are now mostly migration knowledge. Writing custom connectors for common SaaS sources is less central, since managed ingestion tools such as Fivetran and Airbyte cover many of them. Memorising syntax also counts for less when assistants are a keystroke away, so spend interview time on design and debugging rather than recall.
Data engineer skills matrix by level
| Skill area | Junior | Mid-level | Senior |
|---|---|---|---|
| SQL and modelling | Correct joins and aggregations; understands grain | Designs dimensional models for a domain | Sets modelling standards across teams |
| Python | Scripts with logging and error handling | Tested, reusable ingestion code | Shared libraries and frameworks for the team |
| Spark and processing | Runs and edits existing jobs | Writes and tunes jobs independently | Chooses the right engine per workload |
| Orchestration | Adds tasks to existing DAGs | Owns schedules, dependencies and backfills | Designs orchestration and recovery patterns |
| Testing and quality | Adds tests when asked | Tests by default, fixes root causes | Sets contracts with source teams |
| Streaming | Understands topics and consumers | Builds pipelines that handle late data | Decides where streaming is worth its cost |
| Governance and access | Follows access rules | Implements masking and permissions | Designs the access model and lineage |
| Communication | Raises problems early | Keeps consumers informed during incidents | Negotiates priorities with data owners |
How to assess data engineer skills in a hiring process
Keyword matching rewards whoever lists the most tools, so build the process around evidence of work. If the role is not yet written down, start with our data engineer job description, then use these steps.
- Ask for a work sample before the interview. A public dbt project, a design document with sensitive details removed, or a short write-up of a pipeline they owned tells you more than a CV.
- Run a lineage exercise. Show a simple diagram of sources, staging tables, models and reports, then say a source column has changed meaning. Ask which tables and reports are affected and who they would tell.
- Allow an AI assistant, then question the output. In a practical task, let them use the tools they would use at work, then ask them to explain the generated query and what they checked. You learn whether they understand the code they ship.
- Ask about the first week on a platform they did not build. Good candidates describe reading lineage, finding the most-used tables, checking failure history and asking consumers what they distrust. Weak ones talk only about rewriting things.
Red flags on a CV: every tool from every cloud with no outcome attached; “big data” experience that stops at Hadoop-era tools; projects built only on public tutorial datasets; no mention of who used the data; and “real time” claimed for pipelines that run hourly.
Frequently asked questions
Does a data engineer need to know machine learning?
Not to build models, but it helps to understand what models need. Engineers who know why training and serving data must match, and how leakage creeps in, make better partners for data scientists.
Is SQL or Python more important for a data engineer?
Both are needed, but SQL usually carries more weight, since most transformation work on modern platforms happens in SQL, often through dbt. Python matters most for ingestion and anything that does not fit neatly into a query.
Can a backend developer move into data engineering?
Often, yes. Backend developers bring good habits in testing, version control and production support. What they usually need to learn is analytical modelling, thinking in sets rather than rows, and the patterns for incremental loads and backfills.
Are cloud certifications a good signal of data engineer skills?
They show someone has studied a platform’s services, which helps when you run that platform. They do not show that the person has kept a pipeline running through real incidents, so treat a certification as a reason to ask deeper questions and weigh delivered work more heavily.
If you want engineers with these skills, Zilo AI can help you hire data engineers in India for contract, contract-to-hire or permanent roles, and covers the rest of the data team through our IT talent hiring service. Talk to our team about the platform you run and the work ahead.
