Apache Hudi: The Definitive Guide (for Duc ka) by Shiyan Xu Prashant Wason Bhavani Sudha Saktheeswaran Rebecca Bilbro

Author:Shiyan Xu, Prashant Wason, Bhavani Sudha Saktheeswaran, Rebecca Bilbro
Language: eng
Format: epub
Publisher: O'Reilly Media, Inc.
Published: 2024-10-02T00:00:00+00:00

Figure 1-6. Illustration to show how Hudi Merge-on-Read tables work.

Figure 1-6 illustrates how different queries work in a Merge-on-Read Hudi table. Let’s just take snapshot queries for now and discuss other query types in later sections. We can see that a snapshot query that runs just after commit time T10 will merge the data from the base file and the append logs for file groups 1 & 2 on-the-fly. For file group 3, only the base file data will be returned. For file group 4, only merged data from the append logs will be returned (since there is no base file). The periodic compaction process on the write side (discussed in Chapter 6) reconciles these changes to produce a new version of the base file that has the latest updates to keep the query performances in check. In Figure 1-6, the periodic compaction process happened at commit time T5 to produce version T5 base file in all relevant file groups.

For the Fraud detection team’s use case described previously, the snapshot query issued at 10:47 am should return all the latest trips that are less than 3 mins in duration and happened in San Francisco city between 10:32 am and 10:47 am. This query would, on-the-fly, merge the updates from log files onto base files as needed from the relevant file groups.

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.