Databricks Auto Loader is a managed feature of Spark that incrementally and efficiently processes new data files as they arrive in cloud storage. It supports JSON and many semi-structured formats and is widely used to handle large-scale ingestion of flexible schemas.
Auto Loader incrementally pulls new JSON or other files from cloud storage and writes them to Delta Lake tables for downstream analytics.