Senior Engineer

We're looking for a Senior Software Engineer, Data Science Infrastructure & Optimization, to join our team! This is a critical technical role where you'll build and scale high-performance data infrastructure that powers our data science and modelling capabilities. You'll own the entire data lifecycle for modelling—from ingestion and transformation to automated scheduling—working with large-scale environmental datasets (like historical and forecasted weather data) that serve our entire organization. You'll enable seamless cross-functional data exchange, collaborate with engineering teams to ensure data flows in the right formats, and create self-service tools that empower data scientists to focus on what they do best: analysis and insights.

What You'll Do

• Own and continuously improve data ingestion and transformation pipelines for large-scale climate and renewables datasets (weather data, renewable generation data), ensuring quality and timely delivery across the business
• Facilitate cross-functional data exchange by ingesting and transforming datasets from other engineering teams and delivering them to stakeholders in their required formats
• Build and deploy self-service infrastructure components (AWS Lambda functions, Glue/Athena tables, computing infrastructure) that make data access and preparation seamless for data scientists
• Govern and manage large datasets across AWS environments, including data versioning and resolving quality issues for internal users
• Design and implement optimization strategies for large-scale data processing and complex modeling tasks, leveraging parallelization and distributed computing tools like Dask for maximum performance and efficiency
• Partner with data scientists to develop new code and scripts, refactoring them into maintainable, efficient, and reusable functions that prevent future bottlenecks
• Create shared code frameworks, templates, and internal libraries that enforce best practices, contribute to company-wide tooling, and accelerate data science workflows
• Define and implement comprehensive data quality assurance processes, including validity checks and proactive diagnosis and resolution of production issues
• Build and maintain robust unit and BDD (Behave) test suites that validate complex transformation and modeling logic
• Mentor data scientists on code structure, effective testing practices, and engineering standards
• Work closely with internal teams and end-users to understand their needs, address technical challenges, and co-design scalable architectural solutions that serve the broader organization

• Experience in Data Engineering or software engineering supporting data science or research teams
• Advanced Python proficiency and expert-level experience with distributed computing frameworks (e.g., PySpark, Dask)
• Strong hands-on experience with AWS services for data processing (Step Functions, Lambda, Batch, S3, Athena)
• Deep knowledge of software engineering best practices, including design patterns, refactoring, infrastructure as code, containerization, and CI/CD pipelines
• Proven experience writing comprehensive test suites (unit, integration, and behavioural/Behave tests)