In our Client's Services group, we foster a strong grassroots culture where problem-solvers and innovators have direct influence on product and business decisions. We are a tight-knit group with a make it work, make it fast, make it pretty mission. Each member of our team is expected to take ownership of initiatives, tackle new problems, master new technologies, and shape the future of our company.
About the Position
The Data Engineer is responsible for processing structured and unstructured data, validating data quality, and developing and supporting data products. The Data Engineer also plays a role in Agile planning, providing advice and guidance, and monitoring emerging technologies. You will design, code, test, correct, and document programs and scripts from agreed-upon specifications, and subsequent iterations, using agreed-upon standards and tools, to achieve a well-engineered result.
As a Data Engineer you will:
- Gather and processes raw, structured, semi-structured, and unstructured data using batch and real-time data processing frameworks.
- Implement and optimize data solutions in enterprise data warehouses and big data repositories, focusing primarily on operating in the cloud.
- Understand and enforce appropriate data master management techniques.
- Ensure data quality and implement tools and frameworks for automating the identification of data quality issues.
- Work with internal and external data providers on data validation, provide feedback, and make customized changes to data feeds and data mappings for analytical and operational use.
- Understand the challenges our analytics organization faces in their day-to-day work, and partner with them to design viable data solutions.
- Provide ongoing support, monitoring, and maintenance of deployed products in a build-and-run model.
- Automate manual processes, transforming them into repeatable capabilities.
- Work directly with stakeholders to understand and solve real world problems in our domain.
- The ideal candidate will have experience with Python, SparkSQL, Java, Groovy, and AWS Services such as Lambda and EMR.
- Experience engineering server-side code for data processing applications, Git or other advanced version control systems.
- Working experience: Hadoop ecosystem technologies, MapReduce-based frameworks, data concepts (SQL, NoSQL, Normalization), complex data structures.
- Exposure to Agile/SAFE methodologies.
- Good technical writing skills including high- and low-level diagramming techniques.
- MS/BS degree in Computer Science, related field, or equivalent work experience.
- Experience with Data Modeling
- Experience with event frameworks (Kafka, Kinesis, RabbitMQ etc.)
- Working experience of statistics and how they apply to operational metrics
- Automotive industry experience