All 7 AWS Glue & ETL questions
More in AWS Glue & ETL
What is AWS Glue and what stages of the data pipeline process does it help automate? How do Glue jobs use Apache Spark under the hood to perform large scale data transformations? What is AWS Glue Studio, and how does its visual interface simplify building ETL jobs for users less comfortable writing Spark code directly? How does AWS Glue support job scheduling and triggering ETL workflows automatically based on events or a defined schedule? How does the Glue Data Catalog serve as a shared metadata layer across multiple AWS analytics services simultaneously? How does AWS Glue support sensitive data handling through features like the Glue Data Quality and Sensitive Data Detection capabilities?
Advertisement
Back to results for “S3 & Storage”
AWS › AWS Glue & ETL

What is a Glue crawler, and how does it automatically populate the AWS Glue Data Catalog?

Beginner

Answer

A Glue crawler connects to a specified data source, such as an S3 bucket or a database, automatically inspects the structure of the data it finds, and creates or updates corresponding table definitions in the Glue Data Catalog, including inferring column names and data types, saving you from manually defining schema information for every dataset you want to query or process.
aws glue create-crawler --name my-crawler --role AWSGlueServiceRole --database-name my-database --targets '{"S3Targets":[{"Path":"s3://my-bucket/data/"}]}'
Real-world example A data engineering team schedules a Glue crawler to run nightly against a folder of newly arriving log files in S3, automatically detecting any new columns that have appeared in the data and keeping the Glue Data Catalog schema up to date without any manual effort.

Common follow-ups: How often should a Glue crawler be scheduled to run for frequently changing data?;What happens if a crawler encounters inconsistent schemas across different files in the same folder?

Amazon Athena;S3 & Storage
All 7 AWS Glue & ETL questions