Job Title: Data Engineer
Location: Fully Remote, working on an EST schedule
Company: Apex Systems
Job Summary:
We are seeking a highly skilled Data Engineer with a focus on Azure and Databricks to support our business objectives. The ideal candidate will have over 5 years of experience in data engineering, preferably in the commercial airline industry or general aerospace. Experience in manufacturing environments is also welcome.
Key Responsibilities:
* Data Pipeline Design and Implementation: Utilize Azure Data Factory and Databricks to process and analyze flight data, ensuring seamless integration with business intelligence tools.
* Data Lakes and Warehouses: Develop and maintain scalable data lakes and warehouses on Azure, guaranteeing data integrity and accessibility for informed decision-making.
* Cross-Functional Collaboration: Work closely with teams to integrate flight data into business intelligence tools, enhancing decision-making processes and driving business growth.
* Automation and Optimization: Leverage Python and SQL to automate data processing tasks, reducing manual effort and increasing efficiency. Conduct performance tuning and optimization of Spark jobs to improve data processing speed.
* Troubleshooting and Support: Provide technical support and troubleshooting for data-related issues, ensuring minimal downtime and optimal system performance.
Required Qualifications:
* Bachelor's degree in Computer Science, Information Technology, or a related field.
* Over 5 years of experience in data engineering, preferably in the commercial airline sector or general aerospace.
* Strong experience with Azure Data Factory, Azure Data Lake, and Azure Synapse Analytics (formerly Azure SQL Data Warehouse).
* Proficiency in Python and SQL.
* Experience with Databricks and Apache Spark.
Preferred Qualifications:
* Experience with Git, Docker, and Kubernetes.