When a data engineer is doing the job well, almost nobody notices. The morning sales dashboard refreshes on time, the churn model gets yesterday’s events, and finance stops asking why two reports show different revenue. The data engineer builds and runs the pipelines that move data from source systems into a warehouse or lakehouse, cleans and shapes it on the way, and keeps those pipelines working as volumes grow and source systems change underneath them.
Below is a ready-to-copy data engineer job description, written so it also works for a big data engineer role, followed by guidance on levels, the skills worth testing, the job ad mistakes that cost you good applicants, and when contract or permanent hiring fits best.
Data engineer job description template
Fill in the square brackets with your own stack and details. If your platform runs on one cloud, name it; specific ads attract specific applicants.
About the role
[Company] is hiring a Data Engineer to build and run the data platform behind our [analytics / product / ML] work. You will own batch and streaming pipelines that bring data from [source systems, such as the product database, CRM and event stream] into our [Snowflake / Databricks / BigQuery / Redshift] platform, and make sure analysts, data scientists and reporting teams can rely on what they find there. You will be based in [city / remote] and report to [manager title].
Responsibilities
- Design, build and maintain batch and streaming ingestion pipelines from [databases, APIs, files and event streams].
- Write and optimize transformations in SQL and [PySpark / dbt] to produce clean, documented, well-modeled tables.
- Orchestrate and schedule workloads with [Airflow / Azure Data Factory / Cloud Composer / Databricks Workflows], with sensible retries and alerting.
- Build data quality checks for freshness, completeness and accuracy, and act on failures before downstream users notice.
- Tune Spark jobs and warehouse queries for performance and cost as data volumes grow.
- Run streaming pipelines with [Kafka / Kinesis / Pub/Sub] where near real-time data is needed.
- Manage access, secrets and permissions on the platform in line with [company] security and governance policies.
- Define infrastructure and pipeline configuration as code, using [Terraform] and version control with code review.
- Work with analysts, data scientists and BI developers to understand what data they need and in what shape.
- Document pipelines, table definitions and on-call runbooks so the platform does not depend on any one person.
Requirements
- [X]+ years of experience building production data pipelines.
- Strong SQL, including window functions, query plans and performance tuning on large tables.
- Good Python for data work, and hands-on experience with Apache Spark or PySpark.
- Experience with at least one orchestration tool such as Airflow, and with dbt or a similar transformation framework.
- Working knowledge of a cloud data platform such as [AWS Glue and Redshift / Azure Data Factory and Synapse / BigQuery / Snowflake / Databricks].
- Understanding of data modeling for analytics, including dimensional models and slowly changing dimensions.
- Comfort with Git, code review and CI for data code.
Nice to have
- Experience with Kafka or another streaming platform in production.
- Experience migrating from a legacy platform such as Teradata, Oracle or Hadoop and Hive.
- Familiarity with Delta Lake, Unity Catalog or Microsoft Fabric.
- A cloud data engineering certification from [AWS / Microsoft / Google / Snowflake / Databricks].
What we offer
- [Salary range] and [benefits].
- [Work mode: on-site / hybrid / remote] and [on-call arrangement, if any].
- [Training or certification budget].
How the data engineer role changes by level
Junior data engineer. A junior engineer adds new sources and transformations to a platform that already exists, following patterns set by others. The job description should focus on writing reliable SQL and Python, fixing failed jobs and learning the orchestration and testing conventions. Ask for strong SQL and some hands-on pipeline work, even from a course project, rather than years of cloud experience.
Mid-level data engineer. This person owns whole pipelines and domains end to end: they choose how a source is ingested, model the resulting tables, add quality checks and handle incidents. Requirements should mention performance tuning, incremental loads and working directly with the teams who consume the data.
Senior data engineer or big data engineer. A senior engineer shapes the platform itself. Expect them to choose between batch and streaming for a use case, plan a migration off a legacy warehouse, set standards for testing and code review, and keep compute costs under control. For big data roles, add experience with very large Spark workloads, partitioning strategy and file layout. Where the work moves into designing the whole warehouse or lakehouse and its governance, you are describing a data architect, and that deserves its own job description.
Data engineer skills that matter most, and how to check them
- SQL under realistic conditions. Give the candidate a small dataset with late-arriving records and duplicate keys, and ask for a deduplicated daily table. Watch how they handle the edge cases, not just whether the query runs.
- Pipeline design. Ask them to sketch how they would load a large table from a production database every night without overloading it. Good answers cover incremental loads, change data capture, idempotent reruns and what happens when a run fails halfway.
- Spark and performance. Ask about a slow job they fixed. Strong candidates mention specifics such as data skew, shuffle size, partition counts or small files, and how they measured the improvement.
- Data quality. Ask how they would find out that a source system silently started sending nulls in a key column. Look for automated tests and alerts, not a promise to check manually.
- Cost awareness. On cloud platforms, ask how they would bring down warehouse or cluster spend on a pipeline that has grown expensive. This separates people who have run a platform from people who have only built on one.
A practical exercise works best when it resembles your real stack: a small dbt project, an Airflow DAG to review, or a PySpark job with a planted performance problem. Reviewing someone else’s code is often more revealing than writing new code.
Common data engineer job description mistakes
- Naming every cloud at once. Asking for deep AWS, Azure and GCP experience in one ad either scares off strong specialists or invites inflated CVs. Fix: name the platform you run, and list the others as nice to have.
- Confusing the role with a data scientist or BI developer. Some ads ask a data engineer to build ML models and Power BI dashboards as well. Fix: keep the focus on pipelines and the platform, and describe modeling or reporting work only if it truly belongs to the role.
- Hiding the on-call expectation. Pipelines fail at night and during month-end. Fix: say plainly whether there is an on-call rota and how it works.
- Leaving out the current state of the platform. A greenfield lakehouse build and the upkeep of an old on-premise warehouse are very different jobs. Fix: describe where the platform is today and what the next project is.
- Treating Hadoop experience as a must. Unless you still run Hadoop, it narrows the pool for no benefit. Fix: ask for Spark and modern platform experience, and mention Hadoop and Hive only if you are migrating off them.
Contract, contract-to-hire or permanent: which suits a data engineer
Contract data engineers are a strong fit for migrations and builds with a clear finish line, such as moving off an on-premise warehouse, setting up a first lakehouse, or adding a streaming pipeline for a new product. You can bring in people who have done the same migration before, on the exact services you use, and let the engagement end once the work is handed over.
Contract-to-hire works well when you are forming a data team and want to see how someone handles your real sources and incidents before making a long-term offer. A data platform also builds up a lot of undocumented knowledge, so a trial period helps you judge whether the person writes things down.
Permanent hiring suits the steady work of running and growing the platform, where knowledge of your sources, quirks and consumers pays off over time. Zilo AI can support all three routes: we hire data engineers in India on contract, contract-to-hire or permanent terms, and shortlist by the exact services you run, from Spark and Airflow to Snowflake, Databricks and BigQuery. For the wider team, including data scientists and BI developers, see our IT talent hiring page.
Frequently asked questions
What should a data engineer job description include?
It should name the data platform and main tools, describe the sources and pipelines the person will own, state the reporting line and any on-call duty, and list the core requirements in SQL, Python, Spark and orchestration. Add the current state of the platform so applicants understand the job.
What is the difference between a data engineer and a big data engineer?
The work is the same in kind. A big data engineer job description puts more weight on distributed processing at very large volumes, usually with Spark, Kafka and careful partitioning and file layout. Many companies now use the plain data engineer title for both.
What is the difference between a data engineer and a data scientist?
Looking at it from the engineering side, the data engineer is responsible for data being correct, on time and easy to query. The data scientist uses that data to build models and run experiments. If your data scientists spend most of their week fixing broken loads, you need a data engineer.
Should a data engineer job description ask for a specific cloud?
Yes, if your platform runs on one. Naming the actual services, such as Glue and Redshift or Data Factory and Fabric, helps the right people recognize themselves in the ad. Strong engineers can learn a second cloud, so keep the others as nice to have.
If you would prefer a shortlist of data engineers who already know your stack, Zilo AI can run the search and screening for you. Contact our team with a short note about your platform and the work you have planned.