Position Details:

Title: Big Data Engineer

Location: Chicago, IL- Hybrid - 3x/week onsite

Compensation: $65-$68.26/hour 

Interview Process: One Inperson round

Why open: New project 

Scope of role: Experience with Kafka, Pyspark and real-time data processing, Elastic Search, Kibana (nice to have)

Project: Dealing with streaming data

Expected to accomplish: 

• Processing Data in Real-time
• Building system from scratch
• Designing abstract layers

Must have: 

• Strong communication skills
• Kafka
• Pyspark
• Real time data processing
• SQL
• Spark 
• Coding/ development in Python

Nice to have: 

• Elastic Search
• Kibana
• Designing system from scratch/ designing abstract layers

Required Skills:

• Strong SQL Skills – one or more of MySQL, HIVE, Impala, SPARK SQL
• Data ingestion experience from message queue, file share, REST API, relational database, etc. and experience with data formats like json, csv, xml
• Experience working with SPARK Structured steaming
• Experience working with Hadoop/Big Data and Distributed Systems
• Working experience with Spark, Sqoop, Kafka, MapReduce, NoSQL Database like HBase, SOLR, CDP or HDP, Cloudera or Hortonworks, Elastic Search, Kibana, etc.
• Hands on programming experience in at least one of Scala, Python, PHP, or Shell Scripting, to name a few
• Performance tuning experience with spark /MapReduce or SQL jobs
• Experience and proficiency with Linux operating system is a must
• Experience in end-to-end design and build process of Near-Real Time and Batch Data Pipelines
• Experience working in Agile development process and deep understanding of various phases of the Software Development Life Cycle
• Experience using Source Code and Version Control systems like SVN, Git, Bit Bucket etc.
• Experience working with Jenkins and Jar management
• Self-starter who works with minimal supervision and the ability to work in a team of diverse skill sets
• Ability to comprehend customer requests and provide the correct solution
• Strong analytical mind to help take on complicated problems
• Desire to resolve issues and dive into potential issues
• Ability to adapt and continue to learn new technologies is important