Senior Data Engineer

Basic Function

The Senior Data Engineer is responsible for designing, building, and optimizing the foundational data infrastructure and tools that support Lumin Digital’s data ecosystem. With a strong focus on the Databricks data platform, this role develops scalable, reliable, and cost-effective solutions that enable engineering and analytics teams to accelerate development and delivery.

The Senior Data Engineer collaborates with cross-functional stakeholders to identify platform needs, develop integrations, improve performance, and implement reusable tools and capabilities that enhance developer productivity.

This role also contributes to technical planning, architectural decisions, and continuous improvement initiatives to ensure the data platform remains scalable, sustainable, secure, and aligned with evolving business needs.

Essential Functions and Responsibilities

• Design, develop, and maintain scalable, reliable, and cost-effective data platform architecture and foundational capabilities, with a primary focus on Databricks.
• Build and optimize data pipelines, integrations, and reusable tools that streamline data processing, improve developer productivity, and accelerate delivery across engineering and analytics teams.
• Monitor and optimize data platform performance, reliability, and resource utilization, identifying opportunities to improve efficiency and reduce operational costs.
• Collaborate with security and engineering teams to ensure data platform solutions align with established security standards, data governance practices, and applicable regulatory and compliance requirements.
• Partner with cross-functional stakeholders to identify shared platform needs, prioritize foundational improvements, and contribute to technical planning and architectural decisions.
• Perform other duties as assigned.

Physical Demands

• While performing the duties of this Job, the employee is regularly required to sit; use hands to type, handle, or feel and talk or hear
• Specific vision abilities required by this job include close vision
• Ability to occasionally lift/move up to 25 pounds
• Individuals with a disability who are otherwise able to perform the essential functions of the job may request reasonable accommodation through the Human Resources department.

Position Specifications

Education

• Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience.

Experience

• 5+ years of experience in software engineering, data engineering, or related technical disciplines.
• 5+ years of hands-on Databricks experience designing, developing, administering, and maintaining scalable cloud-based data platforms.
• Experience building data pipelines, integrations, automation, and reusable tools that support data processing and improve engineering productivity.
• Experience configuring, monitoring, troubleshooting, and optimizing data platform infrastructure for performance, reliability, scalability, and cost efficiency.
• Experience working with cloud infrastructure and services, preferably AWS, with experience in Azure or Google Cloud also valued.
• Experience developing and maintaining solutions using Scala, Python, SQL, Apache Spark, or similar data engineering technologies.
• Experience implementing automated infrastructure provisioning, configuration management, and deployment processes using infrastructure-as-code and CI/CD practices.
• Experience supporting production environments, resolving complex technical issues, and implementing improvements that increase platform stability and operational efficiency.
• Experience collaborating with cross-functional engineering teams to deliver shared technical capabilities and platform improvements.
• Familiarity with streaming data platforms and event-driven architectures, such as Apache Kafka or Confluent, is preferred.

Knowledge, Skills, & Abilities

• Demonstrates humility, integrity, accountability, and a strong sense of ownership. Willing to collaborate across teams, take a hands-on approach to solving challenges, and contribute wherever needed to achieve shared outcomes.
• Commitment to continually improving yourself.
• Strong communication and collaboration skills, with the ability to document technical solutions, share knowledge, and contribute to engineering standards and best practices.
• Strong understanding of modern data platform architecture, distributed data processing, and cloud-based infrastructure.
• Knowledge of Databricks platform capabilities, including workspace administration, compute configuration, workflows, access management, monitoring, and cost optimization.
• Strong programming and scripting skills, particularly in Python and SQL, with the ability to develop reliable, maintainable, and reusable solutions.
• Understanding of Apache Spark, Delta Lake, and data processing and storage optimization techniques.
• Knowledge of cloud infrastructure, security, networking, and resource management, particularly within AWS environments.
• Ability to design and implement platform integrations, automation, and foundational tools that simplify development and accelerate delivery for other engineering teams.
• Understanding of infrastructure-as-code, CI/CD, and automated deployment practices.
• Knowledge of data governance, security best practices, access controls, and platform reliability principles.

Travel

• Minimal, generally 12 days or less per year, ~2X team get togethers a year

Benefits include

We take care of our people with medical, dental, and vision insurance, a 401(k) with company match, flexible PTO plus 12 paid holidays, paid sick leave, and paid parental and family leave. We also offer a lifestyle spending account, tuition reimbursement, and a cell phone stipend. Additional details are provided during the interview process.