Lead Data Engineer
- business Antal International
- directions_car Gurgaon
- workFull-time
Desired skills: ·
A university degree from a reputed University with a record of academic excellence in Mathematics, Computer Science, Engineering, Physics, or similar. ·
8+ years of relevant experience in Data Engineering, warehousing, ETL, automation, big data and cloud technologies, etc. ·
Ability to write clean, scalable, maintainable code in Python.
Has a good understanding of software engineering concepts and patterns.
Proficiency in other languages like Scala, Java, C#, C++ are an advantage. ·
Proven record of building and maintaining data pipelines deployed in at least one of the big 3 cloud ML stacks (AWS, Azure, GCP). ·
Hands-on experience with:
o Open-source ETL, and data pipeline and orchestration of such tools as Apache Airflow and Nifi.
o Large Scale/ Big Data technology, such as Hadoop, Spark, Hive, Impala, PrestoDb, Kafka.
o Fast and distributed data processing in Ray, Dask, DuckDb, Polars, cuDF
o Building performant code in C, C++ and interfaces with python in PyO3.rs, Cython
o Workflow orchestration tools like Apache Airflow.
o Containerisation with Docker, deployment on Kubernetes.
o NoSql and graph databases.
o Unix server administration and shell scripting. ·
Experience in building scalable data pipelines for highly unstructured data. ·
Experience in building DWH and data lakes architectures. ·
Experience in working in cross-functional team with software engineers, data scientists and machine learning engineers. ·
Experience in working with or leading an off-shore team. ·
Proven record of building data science environments deploying ML solutions in at least one of the big 3 cloud ML stacks (Azure/Aws/GCP) and on Kubernetes clusters. ·
Excellent written and verbal command of English. ·
Strong problem-solving, analytical, and quantitative skills. ·
A professional attitude and service orientation with the ability to work with our international teams
Place of work
Gurgaon
India
Employer profile
In 1993, a visionary in London set out to create a better way to connect talented individuals with job opportunities. Fast forward 30 years, and that vision has grown into a worldwide network of over 800 consultants spanning 32 countries. As one of the top recruitment companies, we specialize in IT, Accountancy, Sales and Marketing, Engineering, and more, offering game-changing recruitment consultancy and talent acquisition services to companies of all sizes. Join us on this journey of growth! With our personalized approach to the hiring process, we aim to make finding the right job a positive and stress-free experience for you as a candidate. We understand that job searching can be overwhelming, so we offer our expertise every step of the way to help you navigate the process with ease. Our goal is to empower you to achieve your career aspirations and land the perfect job! At our core, we believe that our success is directly tied to the success of the candidates we work with!
Local radius
- Najafgarh
- Farrukhnagar
- Manesar
- Gurugram
- Shahdara
Job ID: 8443732 / Ref: d25cab61376396f919797adae72e9c30