Is there any testing tools which can help in automating spark jobs and verification of hive tables

neel shah's picture
neel shah asked on August 18, 2020 - 11:52am | Replies (1).

im currently facing a challenge i need to automate spark jobs ( basically spark jobs trigger the job which takes data from input tables and put into output tables ...these are hive tables)  is there any tool or way to automate end to end effectively.....thanks in advance..any approaches are welcome

1 Answer

Milla Ahonen's picture

Apache Oozie is a great tool that will perfectly suit your purpose of automating spark jobs.

StickyMinds is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.