Big Data Architect
Location: Pittsburgh, PA
Job Type: Full Time / Permanent
The Big Data Architect leads the architecture and design of company’s Data Platform in alignment with our vision of providing a unified data solution: aggregating customer and healthcare operational data, and providing easy access to powerful insights. As such, this position will be responsible for the technical and security architecture of the software applications, as well as the supporting infrastructure. This position will help drive quality, reliable, secure applications, using industry standard best practices.
- Lead a development team of big data designers, developers, data scientists, and DevOps
- Implement a big data enterprise warehouse, BI and analytics system using Hive, Spark, RedShift, EMR (Hadoop), and S3
- Develop and maintain processes to acquire, analyze, store, cleanse, and transform large datasets using tools like Spark, Kafka, Sqoop, Hive, NiFi, and MiNiFi
- Provide recommendations, technical direction and leadership for the selection and incorporation of new technologies into the Hadoop ecosystem
- Participate in regular status meetings to track progress, resolve issues, mitigate risks and escalate concerns in a timely manner
- Contribute to the development, review, and maintenance of product requirements documents, technical design documents, and functional specifications
- Help design innovative, customer-centric solutions based on deep knowledge of large-scale, data-driven technology and the healthcare industry
- Help develop and maintain enterprise data standards, best practices, security policies and governance processes for the Hadoop ecosystem
Education & Experience:
- Bachelor’s in computer information technology, computer science, management systems or related discipline required; Master’s preferred
- Four-year degree in Computer Science/Software Engineering or related degree program, or equivalent application development, implementation and operations experience.
- Advanced study or degrees such as Master’s degree in Business (MBA), Masters, PhD., in Computer Science/Software Engineering or a related scientific degree program is preferred
- Minimum 5+ years of experience in large systems analysis and development, addressing unique issues of architecture and data management. Has the experience to work at the highest technical level of all phases of systems analysis and development activity, across the full scope of system development cycle
- 4+ years related experience on data warehousing and business intelligence projects
- 3+ years implementation or development experience with the Hadoop ecosystem
- Working knowledge of the entire big data development stack
- Experience handling very large data sets (10’s of terabytes and up preferred)
- Experience with secure RESTful Web Services
- Highly proficient with Java/Scala application development
- Expert in Apache Spark infrastructure and development
- Experience with Sqoop, Spark, Hive, Kafka, and Hadoop
- Experience with automated testing for Big Data platforms
- Experience with best practices for data integration, transformation, governance and data quality
- Experience with developing, designing and coding, completing programming and documentation, and performing testing for complex ETL applications (Spark & Scala preferred)
- Experience with Agile software development process and development best practices
- Experience with Big Data text mining and big Data Analytics preferred
- Understanding of Big Data Architecture along with tools being used on Hadoop ecosystem
- Ability to lead tool suite selection and lead proofs of concepts
- Ability to share ideas among a collaborative team and drive the team based on technical expertise and sharing best practices from past experience
- Strong understanding and experience executing several software development methodologies and life cycles. Ability to understand and translate business requirements into technical specifications.
- Ability to negotiate with and influence senior management. Ability to lead and influence across departments and across levels of leaderships both internally and with customers.
- Proven ability to organize/manage multiple priorities coupled with the flexibility to quickly adapt to ever-changing business needs.
- Excellent written and oral communication skills. Adept and presenting complex topics, influencing and executing with timely / actionable follow-through.
- Strong analytical and problem-solving skills with the ability to convert information into practical training deliverables. Uses rigorous logic and methods to solve difficult problems.