Principal Engineer, MQ & Kafka

About this role:

Wells Fargo is seeking a deeply technical Principal Engineer with elite-level expertise in both IBM MQ and Apache Kafka. This is a hands-on-keyboard role for a subject matter expert who will be the ultimate technical authority for our enterprise messaging and data streaming backbone. You will be responsible for architecting, building, securing, and optimizing our most critical data-in-motion platforms to support high-volume, low-latency financial applications. This is a role for a master engineer who solves the most complex distributed systems challenges.

In this role, you will:

Architecture & Engineering

• Architect, build, and optimize enterprise-grade IBM MQ and Apache Kafka infrastructure from the ground up.
• Design and implement resilient, high-availability (HA) and disaster recovery (DR) topologies, including MQ Multi-Instance Queue Managers/Clusters and Kafka cluster replication (e.g., MirrorMaker2).
• Engineer solutions for diverse messaging patterns: request/reply, pub/sub, transactional, and event streaming.
• Define and enforce enterprise standards for MQ queue/channel definitions, Kafka topic naming conventions, partitioning strategies, and data schemas (using Avro/Protobuf and Schema Registry).
• Serve as the technical design authority for all projects integrating with MQ or Kafka.

Implementation & Administration

• Perform expert-level installation, configuration, and tuning of IBM MQ (Queue Managers, Channels, Listeners) and Kafka (Brokers, Zookeeper/KRaft, Connect).
• Implement advanced security controls: TLS/SSL for both platforms, Channel Authentication (CHLAUTH) and OAM in MQ, and SASL/SCRAM with ACLs in Kafka.
• Develop and maintain a robust automation framework (using Ansible, Python, Terraform) for provisioning, configuration management, and operational tasks for both MQ and Kafka.
• Manage and optimize the Kafka Connect ecosystem, deploying and monitoring connectors for data integration.

Performance & Troubleshooting

• Lead performance tuning efforts to maximize throughput and minimize latency for both MQ and Kafka, focusing on buffer tuning, batching, compression, and log management.
• Conduct deep-dive root cause analysis (RCA) for production incidents, analyzing FDC files and error logs in MQ, and broker/consumer logs and metrics in Kafka.
• Utilize advanced debugging tools (e.g., tcpdump, Wireshark, JVM profilers) to diagnose complex network, application, and platform issues.
• Proactively monitor platform health, consumer lag, message throughput, and system resource utilization using tools like Prometheus, Grafana, and enterprise monitoring suites.

Developer & Application Support

• Act as a senior consultant to application development teams on best practices for using MQI, JMS, and Kafka Producer/Consumer APIs.
• Troubleshoot critical integration issues, including poison messages, stuck consumers, message ordering conflicts, and idempotent producer problems.
• Champion the adoption of modern practices like event-driven architecture and stream processing (using Kafka Streams or ksqlDB).

Required Qualifications:

• 7+ years of Engineering experience, or equivalent demonstrated through one or a combination of the following: work experience, training, military experience, education

Desired Qualifications:

• deep, hands-on engineering experience with both IBM MQ and Apache Kafka in a large-scale enterprise environment.
• IBM MQ Expertise: Mastery of MQSC commands, MQ Explorer, and architectural patterns (Clustering, Multi-Instance). Deep knowledge of MQ security (OAM, CHLAUTH) and log management.
• Kafka Expertise: Mastery of the Kafka ecosystem (Brokers, Zookeeper/KRaft, Connect, Schema Registry). Proven experience with Kafka security (SASL, ACLs, mTLS) and performance tuning.
• Automation Proficiency: Strong scripting and automation skills using Ansible, Python, Shell, or Terraform are essential.
• Integration Knowledge: Expert-level understanding of JMS, MQI, and Kafka client APIs.
• Troubleshooting: Elite-level debugging skills with the ability to analyze everything from network packets to application code and system logs.
• Operating Systems & Networking: Solid expertise in Linux/UNIX and a strong understanding of TCP/IP, firewalls, and load balancers as they relate to distributed messaging systems.
• High-Volume Environments: Experience in financial services or another industry with high-throughput, low-latency, and zero-data-loss requirements is a major plus.
• A Bachelor's degree in Computer Science/Engineering or equivalent real-world experience.
• A builder's mentality with a passion for automation and infrastructure-as-code.
• An obsession with performance and reliability.
• The ability to remain calm and methodical while troubleshooting high-pressure production outages.
• A natural collaborator who enjoys mentoring developers and other engineers.

Job Expectations:

• This role is not eligible for visa sponsorship

Posting End Date:

*Job posting may come down early due to volume of applicants.

We Value Equal Opportunity

Wells Fargo is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other legally protected characteristic.

Employees support our focus on building strong customer relationships balanced with a strong risk mitigating and compliance-driven culture which firmly establishes those disciplines as critical to the success of our customers and company. They are accountable for execution of all applicable risk programs (Credit, Market, Financial Crimes, Operational, Regulatory Compliance), which includes effectively following and adhering to applicable Wells Fargo policies and procedures, appropriately fulfilling risk and compliance obligations, timely and effective escalation and remediation of issues, and making sound risk decisions. There is emphasis on proactive monitoring, governance, risk identification and escalation, as well as making sound risk decisions commensurate with the business unit’s risk appetite and all risk and compliance program requirements.

Candidates applying to job openings posted in Canada: Applications for employment are encouraged from all qualified candidates, including women, persons with disabilities, aboriginal peoples and visible minorities. Accommodation for applicants with disabilities is available upon request in connection with the recruitment process.

Applicants with Disabilities

To request a medical accommodation during the application or interview process, visit Disability Inclusion at Wells Fargo .

Drug and Alcohol Policy

Wells Fargo maintains a drug free workplace. Please see our Drug and Alcohol Policy to learn more.

Wells Fargo Recruitment and Hiring Requirements:

a. Third-Party recordings are prohibited unless authorized by Wells Fargo.

b. Wells Fargo requires you to directly represent your own experiences during the recruiting and hiring process.