Role
Summary:
This
job involves creating data pipelines using a configurable framework
(developed using Databricks / pyspark), which are used to load data from
multiple sources to Google Cloud. It also involves extensive collaboration
with other team members (India & USA) to follow a plan, execute
workflows, ensure data load accuracy and debug in case of issues.
Essential
Responsibilities
•
Design, develop, and implement robust, high-performance data pipelines using
Databricks/PySpark to meet diverse data integration requirements. Understand
data and its relation / significance from business perspective
•
Leverage extensive SQL expertise for complex data manipulation, analysis, and
query optimization.
•
Experience in Google Cloud services like Dataproc, BigQuery, Airflow, etc.
•
Create custom scripts in python / pyspark
•
Minor development and enhancement activities as needed to handle new / custom
requirements
•
Validate data between source and target & highlight issues and rerun jobs
as needed
•
Lead other developers, review their code and ensure quality & optimal
code is being delivered
•
Provide regular updates on the tasks, status and risks to project manager
The
experience we are looking to add to our team
Required
•
Bachelor’s degree from a reputed university
•
7 to 12 years overall experience
•
Excellent SQL / data knowledge (6+ years) in creating complex SQL queries,
Stored Procedures etc.
•
Experience in developing enterprise grade ETL applications (any tool)
•
Experience in Google BigQuery or any cloud database
•
Experience in Google Cloud services like Managed service for Apache
Spark(Dataproc), Airflow etc.
•
Working knowledge in Pyspark / Python
•
Good communication and positive attitude
•
Attitude to take ownership and drive tasks without follow-ups
Good
to have
•
Experience in Big data technologies like Hadoop, Hive, etc.
•
Knowledge in Iceberg, Starburst, DBT
•
Knowledge in US Healthcare domain
•
Knowledge in Agile