Data Engineer (Kafka)
About the Role
In the world of transportation, data is constantly moving, and Kafka is the roadway that keeps that traffic running smoothly to its destination. As a technical expert, you must be comfortable working across teams on multiple, high impact projects. You will be a valued part of a team that is constantly maturing Kafka use and event-driven architecture. Members of this team are responsible for the overall use and implementation of Kafka components including the Confluent platform, observability, governance, best practices, and solution development. An understanding of Kafka principles and enterprise integration patterns is required.
In order to be successful:
- You are a self-directed person who can identify priorities.
- You are a detail-oriented person who takes pride in keeping data correct and always having a backup plan.
- You are a problem-solver who might write a script or find a tool to get things done when there isn't an established solution.
- You want to learn and grow in the event-driven world.
- You love Kafka! When you hear terms like "event-driven" or "real-time streaming" you're ready to dive in!
Responsibilities
- Develop and implement solutions using Kafka.
- Administer and improve use of Kafka across the organization including Kafka Connect, ksqlDB, Streams, and custom implementations.
- Work with multiple teams to ensure best use of Kafka and data-safe event streaming.
- Understand and apply event-driven architecture patterns and Kafka best practices. Enable development teams to do the same.
- Assist developers in choosing correct patterns, event modeling, and ensuring data integrity.
- Continuous learning to be a Confluent/Kafka subject matter expert.
- Work with Kafka and Confluent API's (e.g. metadata, metrics, admin) to provide pro-active insights and automation.
- Work with SRE's to ensure Kafka-related metrics are exported to New Relic.
- Perform regular reviews of performance data to ensure efficiency and resiliency.
- Contribute regularly to event-driven patterns, best practices, and guidance.
- Review feature release and change logs for Kafka, Confluent, and other related components to ensure best use of these systems across the organization.
- Work with lead to ensure all teams are aware of technology changes and impact.
- Develop an expert-level understanding of data migration and CDC as it relates to Kafka using Kafka Connect and Debezium.
- Acquire a deep understanding of source and sink connector technical details for a variety of platforms including PostgreSQL, MS SQL Server, Snowflake, and others as required.
Requirements
- Be able to describe the primary components of Kafka and their function (brokers, zookeeper, topics).
- At least two years of experience supporting applications in a production environment.
- You will be expected to read and navigate code in multiple languages. Multi-language fluency and writing is not required.
- Proficiency in at least one programming language and one scripting language.
- Proficiency with Docker containers.
- Ability to participate in and contribute to code management in GitHub including actively collaborating in peer-reviews, feature branches, and resolving conflicts and commits.
- Excellent written and verbal communication skills.
- Strong sense of responsibility with a bias towards action.
- Comfortable self-directing and prioritizing your own work.
- Microservices experience is a plus.
- Distributed tracing experience a plus.
- An understanding of any cloud (Azure preferred) infrastructure and components is a plus but is not required.
About Mastery
Mastery Logistics Systems is building the world’s first lovable Transportation Management System, or TMS.
Our customers – large transportation companies and shippers who need those companies – have struggled with systems that are outdated or inadequate. As shippers or transportation service providers, our customers have in the past been forced to use multiple systems to manage dedicated fleet operations, outsourced or insourced trans management, one way trucking, truckload brokerage, LTL, and Intermodal, or to sub-optimize one or more of those functions by attempting to fit it into a TMS that is adequate at another function.
Mastermind TMS allows our customers to bring all of these functions into a single platform, providing flexibility, visibility, control, and efficiency. Today’s unprecedented global supply chain upheavals underscore how important the transportation industry is. We are building a system to allow this industry to work faster, smarter and more efficiently.
The challenges in this industry are big and exciting! We are tackling everything from fast and efficient data input to ingesting large amounts of data and applying AI to looking at blockchain to securely digitize paperwork. If you are passionate about humanizing an industry, automating in innovative ways, building for quality and scale, helping make people's lives easier and touching every part of our economy then this is the place for you.
Mastery Logistics Systems is committed to continuing to build an incredible company. We are a masterful mosaic of incredible people. We are specialists and experienced in our respective fields. We are dedicated to continuous improvement both professionally and personally. We are a collective group of really good people. We have different interests, backgrounds & talents and we work together to create really cool stuff! We believe in diversity of thought and are mindful and inclusive. We have deep respect for each other and work diligently at adding the right people to our teams.
At this moment we are all working from home and doing our part to combat the Covid 19 virus. We are creatively building our new work habits. We are respectful of each other’s time and personal life. We have flexible schedules but share in the mission that we are building and need to get it done. We offer an excellent suite of benefits. We are dedicated to finding new ways to add perks as we live and work from home.
Our team has the domain knowledge and connections to make an impact, and we’re looking for experienced and thoughtful people to who thrive on creating and building great products. We want people who have a true passion for servicing and taking care of our customers. We need people who are flexible problem solvers, thrive on collaboration and consistently know how to communicate their solutions well. We are small and nimble – which is evident in how quickly we could pivot to our new reality. Each member of the team can make a tremendous impact both technically and culturally. While a start-up, we are well-funded, have an initial paying customer with which to test and launch, and are founded by top experts and veterans in the logistics industry.
Join us – you’ll love it – let’s build a masterpiece!
Benefits
Mastery takes great pride in providing our employees a robust and highly competitive benefit package. Our benefits include Medical, Dental and Vision insurance covering 90% of premium costs. Company paid life insurance for 1x salary. Legal, AD&D, Additional Life and other employee assistance benefits. We have a 401k savings plan with a 4% match. We provide opportunities for professional growth and development. We fully support our work from home initiative as we do our part to combat the Covid 19 crisis. We have a manage your life and schedule Paid Time Off program. We are fully devoted to finding creative perks and benefits since we cannot currently enjoy our cool office culture. Our philanthropic partner is St. Jude Children’s Research Hospital.
We are an equal opportunity employer and actively seek a diverse community of professionals. Veterans, Women, non-binary, people of color, LGBTQIA, we welcome all to apply!