Big Data Engineers will own, define, design, help build the infrastructure including making recommendations with on premise/off-premise based solutions to our clients for their data ingestion and processing needs.
You must be a technical, hands-on leader who knows how to scale a real-time analytics and other data platforms.
• Build world class high-volume real-time data ingestion frameworks and automate ingesting various data sources into Hadoop and other data platforms.
• Research, develop, optimize and innovate frameworks and related components for enterprise scale data analysis and computations
• Develop validation frameworks, proactive monitoring solutions to detect data ingestion failures in big data platform and take appropriate remedies
• Develop Data Adapters to ingest large volume of Unstructured, Semi-Structured and Structured data from various data sources and types
• Collaborate with people working on traditional Data Warehouse technologies and ensure consistency for the data exposed through these different channels
• Own the end-to-end development life-cycle with high quality of solution/code you develop and evangelize the test driven development approach
Must have the following qualifications:
• 8+ years of experience with software analysis, design, development and architecture
• Strong Object Oriented programming experience using Java and Scala required
• Hands-on experience scaling and modifying Hadoop ecosystem-including: HDFS, Web HDFS, HBase, Spark, Spark SQL, Spark Streaming, HIVE, PIG, MapReduce, ZooKeeper, Kafka, Cassandra
• Strong knowledge and experience with NoSQL databases (HBase preferred)
• Deep technical understanding of Geo Indexing and clustering
• Prediction analysis and Data Crunching (Machine learning using PredictionIO , Spark MLib ,R or Python)
• Hands on experience with Solr or Elastic Search.
• Experience and the ability to lead development activities using Agile Scrum methodologies
• Experience with RDBMS, O-R mapping and application of distributed caching technologies
• Proponent of and ability to guide development activities using a TDD approach
• API and REST based web services development a plus