Big Data Software Engineer

Job description

DevOps Engineer

Job Responsibilities
Act as subject matter expert and keep the team up-to-date on standards and practices, technology trends in the area of big data
Stay current on big data trends and research various technologies as they become relevant.
Develop solutions for data extraction, preparation, and loading of data from a variety of relational and non-relational sources into the Big Data environment.
Build data and analytics tools that will offer deeper insight into the pipeline, allowing for critical discoveries surrounding key performance indicators
Design, implement, test, deploy and support near real-time and batch data pipelines, using Hadoop infrastructure.
Design scalable and maintainable solutions, using Big Data tools such as Apache Spark, Apache Kafka, and similar
Adopt and enforce best practices related to data ingestion and extraction of data from the big data platform.
Gather project requirements, meeting with project stakeholders and various operational and business teams
Tune performance of Hadoop clusters, Kafka, Spark jobs, Hadoop MapReduce or similar
Working with the BI, Operational systems teams, and EDW team to develop a modern dimensional representation of our transactional data.
Creating the Snowflake database objects (tables, indexes, etc.) required to support the data model
Work with the data warehouse, business intelligence and advanced analytics teams to evaluate their Big Data use cases and provide feedback and guidance
Escalate support issues with internal teams and vendors

Auto req ID

157854BR
Required Education

Bachelor's Degree
Role ( Job Role )

Software Engineer
State / Province

NORTH CAROLINA
Primary job category

Software Development & Support
Company

(0147) International Business Machines Corporation
Contract type

Regular
Employment Type

Full-Time
ERBP

Yes
Is this role a commissionable/sales incentive based position?

No
Travel Required

Up to 10% or 1 day a week
IBM Business Group

F&O
Preferred Education

Master's Degree
City / Township / Village

RESEARCH TRIANGLE PARK
EO Statement

IBM is committed to creating a diverse environment and is proud to be an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, disability, age, or veteran status. IBM is also committed to compliance with all fair employment practices regarding citizenship and immigration status.
Required Technical and Professional Expertise

·  Bachelor's degree in Information Science / Information Technology, Computer Science, Engineering, Mathematics or a related field
·  At least 5 years of experience with development process.
·  At least 3 year (required) or 4 years (preferred) of experience in designing and leading the development of production-grade big data platforms including automated data acquisition, standardization, record matching, and processing using Hadoop, Spark, and Elasticsearch.
·  At least 3 years (required) or 5 years (preferred) of hands-on experience in full-stack development such as Java, JavaScript, React, Node.js, HTML5, CSS, JSON, UNIX/LINUX, etc.
·  At least 1 year (required) or 4 years (preferred) of hands-on experience developing applications using cloud-based platforms/micro-services.
·  Strong scripting experience with Python/Scala
·  Excellent written and verbal communication skills for technical writing and client presentations.
Country/Region

United States
Preferred Technical and Professional Experience

·  Experience in one or more ETL Tools with an understanding of best practices for building and designing ETL/ELT code
·  Understanding of best practices for building an Enterprise Data Lake
·  Experience using Database Modeling Tools: Erwin or other tool
·  Experience with GIT
·  Previous experience of container technologies such as Docker/ Kubernetes is desirable but not essential.
Eligibility Requirements

none
Position Type

Professional
Early Professional Track

Track unaligned
New Collar Role

No