Data Engineer

主要职责

• Process complex financial data – Handle large-scale financial datasets from multiple sources, including banks, payment gateways and processors, ensuring data integrity across different formats and structures.
• Build and maintain complex pipelines – Develop and optimise pipelines that apply intricate business rules, financial calculations and transformations for accurate transaction processing.
• Time-bound and event-driven processing – Design event-driven architectures to meet strict SLAs for transaction processing, settlement and reconciliation with banks and payment partners.
• Enable reporting and AI insights – Structure and prepare data for advanced analytics, reporting and AI-driven insights to improve payment success rates, detect fraud and optimise transaction flows.

您需要携带的物品

• Programming: Strong proficiency in Python and SQL, with additional experience in languages such as Java or Scala.
• Big data technologies: Hands-on experience with frameworks like Spark (PySpark), Kafka, Apache Hudi, Iceberg, Apache Flink or similar tools for distributed data processing and real-time streaming.
• Cloud platforms: Familiarity with cloud platforms like AWS, Google Cloud Platform (GCP), or Microsoft Azure for building and managing data infrastructure.
• Data warehousing and modeling: Strong understanding of data warehousing concepts and data modeling principles.
• ETL frameworks: Experience with ETL tools such as Apache Airflow or comparable data transformation frameworks.
• Data lakes and storage: Proficiency in working with data lakes and cloud-based storage solutions like Amazon S3, Google Cloud Storage, or Azure Blob Storage.
• Version control: Expertise in Git for version control and collaborative coding.
• Performance optimisation: Expertise in performance tuning for large-scale data processing, including partitioning, indexing and query optimisation.

经验和要求

• Bachelor’s degree in Computer Science, Information Technology, or equivalent experience.
• 1–5 years of experience in data engineering, ETL development, or database management.
• Prior experience in cloud-based environments (e.g., AWS, GCP, Azure) is highly desirable.
• Proven experience working with large-scale datasets in production environments, with a focus on performance tuning and optimization.