Data Pipelines Pocket Reference by James Densmore

Author:James Densmore [James Densmore]
Language: eng
Format: epub
Publisher: O'Reilly Media, Inc.
Published: 2021-05-24T16:00:00+00:00

SELECT * FROM orders;

In an incremental extraction, only records from the source table that have changed or been added since the last run of the job are extracted. The timestamp of the last extraction can either be stored in an extraction job log table in the data warehouse, or by querying the maximum timestamp in a last_updated column in the destination table in the warehouse. Using the fictional orders table as an example, the SQL query executed on the source MySQL database will look like this.

SELECT * FROM orders WHERE last_updated > {{ last_extraction_run} };

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.