Full download ebook at ebookmass.

com

Data Ingestion with Python Cookbook: A

practical guide to ingesting, monitoring, and
identifying errors in the data ingestion
process 1st Edition Esppenchutz
https://ebookmass.com/product/data-ingestion-with-
python-cookbook-a-practical-guide-to-ingesting-
monitoring-and-identifying-errors-in-the-data-
ingestion-process-1st-edition-esppenchutz/

Download more ebook from https://ebookmass.com

More products digital (pdf, epub, mobi) instant
download maybe you interests ...

Python Data Cleaning Cookbook - Second Edition Michael

Walker

https://ebookmass.com/product/python-data-cleaning-cookbook-second-
edition-michael-walker/

ebookmass.com

Data Engineering with dbt: A practical guide to building a

cloud-based, pragmatic, and dependable data platform with
SQL Zagni
https://ebookmass.com/product/data-engineering-with-dbt-a-practical-
guide-to-building-a-cloud-based-pragmatic-and-dependable-data-
platform-with-sql-zagni/
ebookmass.com

Exploratory Data Analysis with Python Cookbook: Over 50

recipes to analyze, visualize, and extract insights from
structured and unstructured data Oluleye
https://ebookmass.com/product/exploratory-data-analysis-with-python-
cookbook-over-50-recipes-to-analyze-visualize-and-extract-insights-
from-structured-and-unstructured-data-oluleye/
ebookmass.com

Data-Driven SEO with Python: Solve SEO Challenges with

Data Science Using Python 1st Edition Andreas Voniatis

https://ebookmass.com/product/data-driven-seo-with-python-solve-seo-
challenges-with-data-science-using-python-1st-edition-andreas-
voniatis/
ebookmass.com
Statistical Process Monitoring Using Advanced Data-Driven
and Deep Learning Approaches: Theory and Practical
Applications 1st Edition Fouzi Harrou
https://ebookmass.com/product/statistical-process-monitoring-using-
advanced-data-driven-and-deep-learning-approaches-theory-and-
practical-applications-1st-edition-fouzi-harrou/
ebookmass.com

Beginner's Guide to Streamlit with Python: Build Web-Based

Data and Machine Learning Applications 1st Edition Sujay
Raghavendra
https://ebookmass.com/product/beginners-guide-to-streamlit-with-
python-build-web-based-data-and-machine-learning-applications-1st-
edition-sujay-raghavendra/
ebookmass.com

Data Structure and Algorithms With Python: The Ultimate

Guide Towards Coding John Thomas

https://ebookmass.com/product/data-structure-and-algorithms-with-
python-the-ultimate-guide-towards-coding-john-thomas/

ebookmass.com

(eBook PDF) Intro to Python for Computer Science and Data

Science: Learning to Program with AI, Big Data and The
Cloud
https://ebookmass.com/product/ebook-pdf-intro-to-python-for-computer-
science-and-data-science-learning-to-program-with-ai-big-data-and-the-
cloud/
ebookmass.com

Nonclinical Study Contracting and Monitoring: A Practical

Guide 1st Edition

https://ebookmass.com/product/nonclinical-study-contracting-and-
monitoring-a-practical-guide-1st-edition/

ebookmass.com
Data Ingestion with Python
Cookbook

A practical guide to ingesting, monitoring, and identifying

errors in the data ingestion process

Gláucia Esppenchutz

BIRMINGHAM—MUMBAI
Data Ingestion with Python Cookbook
Copyright © 2023 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted
in any form or by any means, without the prior written permission of the publisher, except in the case
of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information
presented. However, the information contained in this book is sold without warranty, either express
or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable
for any damages caused or alleged to have been caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and
products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot
guarantee the accuracy of this information.

Group Product Manager: Reshma Raman

Publishing Product Manager: Arindam Majumdar
Senior Editor: Tiksha Lad
Technical Editor: Devanshi Ayare
Copy Editor: Safis Editing
Project Coordinator: Farheen Fathima
Proofreader: Safis Editing
Indexer: Sejal Dsilva
Production Designer: Jyoti Chauhan
Marketing Coordinator: Nivedita Singh

First published: May 2023

Production reference: 1300523

Published by Packt Publishing Ltd.

Livery Place
35 Livery Street
Birmingham
B3 2PB, UK.

ISBN 978-1-83763-260-2

www.packtpub.com
 This book represents a lot and wouldn’t be possible without my loving husband, Lincoln, and his
support and understanding during this challenging endeavor. I want to thank all my friends that
didn’t let me give up and always boosted my spirits, along with my grandmother, who always believed,
helped, and said I would do big things one day. Finally, I want to thank my beloved and four-pawed
best friend, who is at peace, Minduim, for “helping” me to write this book.

– Gláucia Esppenchutz
 Contributors

About the author

Gláucia Esppenchutz is a data engineer with expertise in managing data pipelines and vast amounts
of data using cloud and on-premises technologies. She worked in companies such as Globo.com,
BMW Group, and Cloudera. Currently, she works at AiFi, specializing in the field of data operations
for autonomous systems.
She comes from the biomedical field and shifted her career ten years ago to chase the dream of
working closely with technology and data. She is in constant contact with the open source community,
mentoring people and helping to manage projects, and has collaborated with the Apache, PyLadies
group, FreeCodeCamp, Udacity, and MentorColor communities.

I want to thank my patient and beloved husband and my friends. Thanks also to my mentors in the
Python open source community and the DataBootCamp founders, who guided me at the beginning
of my journey.
Thanks to the Packt team, who helped me through some hard times; you were terrific!
About the reviewers
Bitthal Khaitan is currently working as a big data and cloud engineer with CVS Health, a Fortune
4 organization. He has a demonstrated history of working in the cloud, data and analytics industry
for 12+ years. His primary certified skills are Google Cloud Platform (GCP), the big data ecosystem
(Hadoop, Spark, etc.), and data warehousing on Teradata. He has worked in all phases of the SDLC
of DW/BI and big data projects with strong expertise in the USA healthcare, insurance and retail
domains. He actively helps new graduates with mentoring, resume reviews, and job hunting tips in
the data engineering domain. Over 20,000 people follow Bitthal on LinkedIn. He is currently based
out of Dallas, Texas, USA.
Jagjeet Makhija is a highly accomplished technology leader with over 20 years of experience. They are
skilled not only in various domains including AI, data warehouse architecture, and business analytics,
but also have a strong passion for staying ahead of technology trends such as AI and ChatGPT.
Jagjeet is recognized for their significant contributions to the industry, particularly in complex proof
of concepts and integrating Microsoft products with ChatGPT. They are also an avid book reviewer
and have actively shared their extensive knowledge and expertise through presentations, blog articles,
and online forums.
Krishnan Raghavan is an IT professional with over 20 years of experience in the area of software
development and delivery excellence across multiple domains and technology, ranging from C++ to
Java, Python, data warehousing, and big data tools and technologies. Krishnan tries to give back to the
community by being part of GDG – Pune Volunteer Group, helping the team in organizing events.
When not working, Krishnan likes to spend time with his wife and daughter, as well as reading fiction,
non-fiction, and technical books. Currently, he is unsuccessfully trying to learn how to play the guitar.
You can connect with Krishnan at mail to: k r i s h n a n @ g m a i l . c o m or via
LinkedIn: www.linkedin.com/in/krishnan-raghavan

I would like to thank my wife, Anita, and daughter, Ananya, for giving me the time and space to
review this book.
 Visit https://ebookmass.com
now to explore a rich
collection of eBooks and enjoy
exciting offers!
 Table of Contents

Prefacexv

Part 1: Fundamentals of Data Ingestion

1
Introduction to Data Ingestion 3
Technical requirements 4 How to do it… 17
Setting up Python and its environment 4 How it works… 21
See also 22
Getting ready 4
How to do it… 5 Creating schemas 22
How it works… 7 Getting ready 22
There’s more… 7 How to do it… 22
See also 7 How it works… 24
Installing PySpark 8 See also 25

Getting ready 8 Applying data governance in ingestion 25

How to do it… 10 Getting ready 25
How it works… 12 How to do it… 26
There’s more… 12 How it works… 28
See also 12 See also 28
Configuring Docker for MongoDB 13 Implementing data replication 29
Getting ready 13 Getting ready 29
How to do it… 13 How to do it… 29
How it works… 14 How it works… 29
There’s more… 16 There’s more… 30
See also 16
Further reading 30
Configuring Docker for Airflow 16
Getting ready 17
viii Table of Contents

2
Principals of Data Access – Accessing Your Data 31
Technical requirements 31 How to do it… 47
Implementing governance in a data How it works… 48
access workflow 32 There’s more… 49
Getting ready 32 See also 52
How to do it… 33 Managing encrypted files 52
How it works… 34 Getting ready 52
See also 34 How to do it… 53
Accessing databases and data How it works… 54
warehouses34 There’s more… 55
Getting ready 35 See also 56
How to do it… 35 Accessing data from AWS using S3 56
How it works… 37 Getting ready 56
There’s more… 38 How to do it… 59
See also 39 How it works… 62
Accessing SSH File Transfer Protocol There’s more… 63
(SFTP) files 39 See also 63
Getting ready 39 Accessing data from GCP using
How to do it… 41 Cloud Storage 64
How it works… 43 Getting ready 64
There’s more… 43 How to do it… 66
See also 44 How it works… 68
Retrieving data using API There’s more… 70
authentication44 Further reading 70
Getting ready 45

3
Data Discovery – Understanding Our Data before Ingesting It 71
Technical requirements 71 How to do it… 73
Documenting the data discovery How it works… 77
process71 Configuring OpenMetadata 77
Getting ready 72 Getting ready 77
 Table of Contents ix

How to do it… 79 Getting ready 86

How it works… 84 How to do it… 88
There’s more… 86 How it works… 92
See also 86
Further reading 93
Connecting OpenMetadata Other tools 94
to our database 86

4
Reading CSV and JSON Files and Solving Problems 95
Technical requirements 95 How it works… 105
Reading a CSV file 96 There’s more… 106
See also 107
Getting ready 96
How to do it… 96 Using PySpark to read CSV files 108
How it works… 98 Getting ready 108
There’s more… 98 How to do it… 108
See also 99 How it works… 109
Reading a JSON file 99 There’s more… 110
See also 114
Getting ready 100
How to do it… 100 Using PySpark to read JSON files 114
How it works… 100 Getting ready 114
There’s more… 101 How to do it… 115
See also 103 How it works… 116
Creating a SparkSession for PySpark 103 There’s more… 117
See also 117
Getting ready 103
How to do it… 104 Further reading 117

5
Ingesting Data from Structured and Unstructured Databases 119
Technical requirements 119 There’s more… 127
Configuring a JDBC connection 120 See also 129

Getting ready 120 Ingesting data from a JDBC database

How to do it… 124 using SQL 129
How it works… 126 Getting ready 129
x Table of Contents

How to do it… 130 Getting ready 146

How it works… 132 How to do it… 146
There’s more… 133 How it works… 148
See also 133 There’s more… 149
See also 150
Connecting to a NoSQL
database (MongoDB) 134 Ingesting data from MongoDB
Getting ready 134 using PySpark 150
How to do it… 136 Getting ready 150
How it works… 142 How to do it… 151
There’s more… 144 How it works… 153
See also 145 There’s more… 155
See also 156
Creating our NoSQL table in
MongoDB145 Further reading 157

6
Using PySpark with Defined and Non-Defined Schemas 159
Technical requirements 159 How to do it… 169
Applying schemas to data ingestion 160 How it works… 170

Getting ready 160 Ingesting unstructured data with a

How to do it… 162 well-defined schema and format 172
How it works… 163 Getting ready 172
There’s more… 165 How to do it… 172
See also 165 How it works… 174
Importing structured data using a There’s more… 176
well-defined schema 165 See also 176

Getting ready 165 Inserting formatted SparkSession

How to do it… 165 logs to facilitate your work 176
How it works… 167 Getting ready 176
There’s more… 168 How to do it… 176
See also 169 How it works… 178
Importing unstructured data without There’s more… 179
a schema 169 See also 179

Getting ready… 169 Further reading 179

 Table of Contents xi

7
Ingesting Analytical Data 181
Technical requirements 181 How it works… 197
Ingesting Parquet files 182 There’s more… 198
See also 200
Getting ready 182
How to do it… 183 Ingesting partitioned data 200
How it works… 184 Getting ready 200
There’s more… 185 How to do it… 201
See also 185 How it works… 201
Ingesting Avro files 185 There’s more… 203
See also 204
Getting ready 186
How to do it… 186 Applying reverse ETL 204
How it works… 188 Getting ready 204
There’s more… 190 How to do it… 205
See also 190 How it works… 206
Applying schemas to analytical data 191 There’s more… 207
See also 207
Getting ready 191
How to do it… 191 Selecting analytical data for reverse
How it works… 194 ETL207
There’s more… 194 Getting ready 207
See also 195 How to do it… 208
Filtering data and handling common How it works… 209
issues195 See also 210

Getting ready 195 Further reading 210

How to do it… 196

Part 2: Structuring the Ingestion Pipeline

8
Designing Monitored Data Workflows 213
Technical requirements 213 Getting ready 214
Inserting logs 214 How to do it… 214
How it works… 216
xii Table of Contents

See also 217 How it works… 229

There’s more… 229
Using log-level types 217
See also 230
Getting ready 217
How to do it… 217 Logging based on data 231
How it works… 219 Getting ready 231
There’s more… 220 How to do it… 231
See also 221 How it works… 232
There’s more… 233
Creating standardized logs 221
See also 233
Getting ready 222
How to do it… 222 Retrieving SparkSession metrics 234
How it works… 224 Getting ready 234
There’s more… 227 How to do it… 234
See also 227 How it works… 237
There’s more… 241
Monitoring our data ingest file size 227
See also 242
Getting ready 228
How to do it… 228 Further reading 242

9
Putting Everything Together with Airflow 243
Technical requirements 244 How to do it… 257
Installing Airflow 244 How it works… 260
There's more… 262
Configuring Airflow 244 See also 262
Getting ready 244
How to do it… 245 Configuring sensors 262
How it works… 247 Getting ready 262
See also 248 How to do it… 263
How it works… 264
Creating DAGs 248 See also 265
Getting ready 248
How to do it… 249 Creating connectors in Airflow 265
How it works… 253 Getting ready 266
There's more… 254 How to do it… 266
See also 255 How it works… 269
There's more… 270
Creating custom operators 255 See also 270
Getting ready 255
 Table of Contents xiii

Creating parallel ingest tasks 270 Getting ready 274

Getting ready 270 How to do it… 275
How to do it… 271 How it works… 277
How it works… 272 There's more… 278
There's more… 273 See also 279
See also 274
Further reading 279
Defining ingest-dependent DAGs 274

10
Logging and Monitoring Your Data Ingest in Airflow 281
Technical requirements 281 Designing advanced monitoring 304
Installing and running Airflow 282 Getting ready 304
How to do it… 306
Creating basic logs in Airflow 283
How it works… 308
Getting ready 284
There’s more… 309
How to do it… 284
See also 309
How it works… 287
See also 289 Using notification operators 309
Getting ready 310
Storing log files in a remote location 289
How to do it… 312
Getting ready 289
How it works… 315
How to do it… 290
There’s more… 318
How it works… 298
See also 299 Using SQL operators for data quality 318
Getting ready 318
Configuring logs in airflow.cfg 299
How to do it… 320
Getting ready 299
How it works… 321
How to do it… 299
There’s more… 323
How it works… 301
See also 323
There’s more… 303
See also 304 Further reading 324

11
Automating Your Data Ingestion Pipelines 325
Technical requirements 325 Scheduling daily ingestions 326
Installing and running Airflow 326 Getting ready 327
 Visit https://ebookmass.com
now to explore a rich
collection of eBooks and enjoy
exciting offers!
xiv Table of Contents

How to do it… 327 There's more… 340

How it works… 330
Setting up the schedule_interval
There's more… 330
parameter340
See also 331
Getting ready 340
Scheduling historical data ingestion 331 How to do it… 341
Getting ready 331 How it works… 342
How to do it… 332 See also 342
How it works… 335
Solving scheduling errors 343
There's more… 336
Getting ready 343
Scheduling data replication 337 How to do it… 343
Getting ready 337 How it works… 346
How to do it… 338 There’s more… 347
How it works… 339
Further reading 347

12
Using Data Observability for Debugging, Error Handling,
and Preventing Downtime 349
Technical requirements 349 Getting ready 358
Docker images 350 How to do it… 358
How it works… 361
Setting up StatsD for monitoring 351 There’s more… 363
Getting ready 351
How to do it… 351 Creating an observability dashboard 363
How it works… 353 Getting ready 363
See also 354 How to do it… 363
How it works… 369
Setting up Prometheus for storing There’s more… 370
metrics354
Getting ready 354 Setting custom alerts or notifications 370
How to do it… 354 Getting ready 371
How it works… 356 How to do it… 371
There’s more… 357 How it works… 377

Setting up Grafana for monitoring 358 Further reading 378

Index379

Other Books You May Enjoy 388

 Preface
Welcome to Data Ingestion with Python Cookbook. I hope you are excited as me to enter the world
of data engineering.
Data Ingestion with Python Cookbook is a practical guide that will empower you to design and implement
efficient data ingestion pipelines. With real-world examples and renowned open-source tools, this
book addresses your queries and hurdles head-on.
Beginning with designing pipelines, you’ll explore working with and without data schemas, constructing
monitored workflows using Airflow, and embracing data observability principles while adhering
to best practices. Tackling the challenges of reading diverse data sources and formats, you’ll gain a
comprehensive understanding of all these.
Our journey continues with essential insights into error logging, identification, resolution, data
orchestration, and effective monitoring. You’ll discover optimal approaches for storing logs, ensuring
easy access and references for them in the future.
By the end of this book, you’ll possess a fully automated setup to initiate data ingestion and pipeline
monitoring. This streamlined process will seamlessly integrate into the subsequent stages of the Extract,
Transform, and Load (ETL) process, propelling your data integration capabilities to new heights. Get
ready to embark on an enlightening and transformative data ingestion journey.

Who this book is for

This comprehensive book is specifically designed for Data Engineers, Data Integration Specialists, and
passionate data enthusiasts seeking a deeper understanding of data ingestion processes, data flows,
and the typical challenges encountered along the way. It provides valuable insights, best practices, and
practical knowledge to enhance your skills and proficiency in handling data ingestion tasks effectively.
Whether you are a beginner in the data world or an experienced developer, this book will suit you.
It is recommended to know the Python programming fundamentals and have basic knowledge of
Docker to read and run this book’s code.

What this book covers

Chapter 1, Introduction to Data Ingestion, introduces you to data ingestion best practices and the
challenges of working with diverse data sources. It explains the importance of the tools covered in
the book, presents them, and provides installation instructions.
xvi Preface

Chapter 2, Data Access Principals – Accessing your Data, explores data access concepts related to data
governance, covering workflows and management of familiar sources such as SFTP servers, APIs,
and cloud providers. It also provides examples of creating data access policies in databases, data
warehouses, and the cloud.
Chapter 3, Data Discovery – Understanding Our Data Before Ingesting It, teaches you the significance of
carrying out the data discovery process before data ingestion. It covers manual discovery, documentation,
and using an open-source tool, OpenMetadata, for local configuration.
Chapter 4, Reading CSV and JSON Files and Solving Problems, introduces you to ingesting CSV and JSON
files using Python and PySpark. It demonstrates handling varying data volumes and infrastructures
while addressing common challenges and providing solutions.
Chapter 5, Ingesting Data from Structured and Unstructured Databases, covers fundamental concepts
of relational and non-relational databases, including everyday use cases. You will learn how to read
and handle data from these models, understand vital considerations, and troubleshoot potential errors.
Chapter 6, Using PySpark with Defined and Non-Defined Schemas, delves deeper into common PySpark
use cases, focusing on handling defined and non-defined schemas. It also explores reading and
understanding complex logs from Spark (PySpark core) and formatting techniques for easier debugging.
Chapter 7, Ingesting Analytical Data, introduces you to analytical data and common formats for reading
and writing. It explores reading partitioned data for improved performance and discusses Reverse
ETL theory with real-life application workflows and diagrams.
Chapter 8, Designing Monitored Data Workflows, covers logging best practices for data ingestion,
facilitating error identification, and debugging. Techniques such as monitoring file size, row count,
and object count enable improved monitoring of dashboards, alerts, and insights.
Chapter 9, Putting Everything Together with Airflow, consolidates the previously presented information
and guides you in building a real-life data ingestion application using Airflow. It covers essential
components, configuration, and issue resolution in the process.
Chapter 10, Logging and Monitoring Your Data Ingest in Airflow, explores advanced logging and
monitoring in data ingestion with Airflow. It covers creating custom operators, setting up notifications,
and monitoring for data anomalies. Configuration of notifications for tools such as Slack is also covered
to stay updated on the data ingestion process.
Chapter 11, Automating Your Data Ingestion Pipelines, focuses on automating data ingests using
previously learned best practices, enabling reader autonomy. It addresses common challenges with
schedulers or orchestration tools and provides solutions to avoid problems in production clusters.
Chapter 12, Using Data Observability for Debugging, Error Handling, and Preventing Downtime,
explores data observability concepts, popular monitoring tools such as Grafana, and best practices
for log storage and data lineage. It also covers creating visualization graphs to monitor data source
issues using Airflow configuration and data ingestion scripts.
 Preface xvii

To get the most out of this book

To execute the code in this book, you must have at least a basic knowledge of Python. We will use
Python as the core language to execute the code. The code examples have been tested using Python
3.8. However, it is expected to still work with future language versions.
Along with Python, this book uses Docker to emulate data systems and applications in our local
machine, such as PostgreSQL, MongoDB, and Airflow. Therefore, a basic knowledge of Docker is
recommended to edit container image files and run and stop containers.
Please, remember that some command-line commands may need adjustments depending on your local
settings or operating system. The commands in the code examples are based on the Linux command-
line syntax and might need some adaptations to run on Windows PowerShell.

Software/Hardware covered in the book OS Requirements

Python 3.8 or higher Windows, Mac OS X, and Linux (any)

Docker Engine 24.0 / Docker Desktop 4.19 Windows, Mac OS X, and Linux (any)

For almost all recipes in this book, you can use a Jupyter Notebook to execute the code. Even though it
is not mandatory to install it, this tool can help you to test the code and try new things on the code due
to the friendly interface.
If you are using the digital version of this book, we advise you to type the code yourself or access
the code via the GitHub repository (link available in the next section). Doing so will help you
avoid any potential errors related to the copying and pasting of code.

Download the example code files

You can download the example code files for this book from GitHub at https://github.com/
PacktPublishing/Data-Ingestion-with-Python-Cookbook. In case there’s an update
to the code, it will be updated on the existing GitHub repository.
We also have other code bundles from our rich catalog of books and videos available at https://
github.com/PacktPublishing/. Check them out!

Download the color images

We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You
can download it here: https://packt.link/xwl0U
xviii Preface

Conventions used
There are a number of text conventions used throughout this book.
Code in text: Indicates code words in text, database table names, folder names, filenames, file
extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: “Then
we proceeded with the with open statement.”
A block of code is set as follows:

def gets_csv_first_line (csv_file):

    logging.info(f"Starting function to read first line")
    try:
        with open(csv_file, 'r') as file:
            logging.info(f"Reading file")

Any command-line input or output is written as follows:

$ python3 –-version
Python 3.8.10

Bold: Indicates a new term, an important word, or words that you see onscreen. For example, words
in menus or dialog boxes appear in the text like this. Here is an example: “Then, when we selected
showString at NativeMethodAccessorImpl.java:0, which redirected us to the
Stages page.”

Tips or important notes

Appear like this.

Sections
In this book, you will find several headings that appear frequently (Getting ready, How to do it..., How
it works..., There’s more..., and See also).
To give clear instructions on how to complete a recipe, use these sections as follows:

Getting ready
This section tells you what to expect in the recipe and describes how to set up any software or any
preliminary settings required for the recipe.

How to do it…
This section contains the steps required to follow the recipe.
 Preface xix

How it works…
This section usually consists of a detailed explanation of what happened in the previous section.

There’s more…
This section consists of additional information about the recipe in order to make you more knowledgeable
about the recipe.

See also
This section provides helpful links to other useful information for the recipe.

Get in touch
Feedback from our readers is always welcome.
General feedback: If you have questions about any aspect of this book, mention the book title in the
subject of your message and email us at customercare@packtpub.com.
Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen.
If you have found a mistake in this book, we would be grateful if you would report this to us. Please
visit www.packtpub.com/support/errata, selecting your book, clicking on the Errata
Submission Form link, and entering the details.
Piracy: If you come across any illegal copies of our works in any form on the Internet, we would
be grateful if you would provide us with the location address or website name. Please contact us at
copyright@packt.com with a link to the material.
If you are interested in becoming an author: If there is a topic that you have expertise in and you
are interested in either writing or contributing to a book, please visit authors.packtpub.com.
Random documents with unrelated
content Scribd suggests to you:
 The Project Gutenberg eBook of
The Rising Son; or, the Antecedents and
Advancement of the Colored Race
 This ebook is for the use of anyone anywhere in the United
States and most other parts of the world at no cost and with
almost no restrictions whatsoever. You may copy it, give it away
or re-use it under the terms of the Project Gutenberg License
included with this ebook or online at www.gutenberg.org. If you
are not located in the United States, you will have to check the
laws of the country where you are located before using this
eBook.

Title: The Rising Son; or, the Antecedents and Advancement of

the Colored Race

Author: William Wells Brown

Contributor: Alonzo D. Moore

Elijah W. Smith

Release date: March 31, 2021 [eBook #64971]

Language: English

Credits: hekula03, Martin Pettit and the Online Distributed

Proofreading Team at https://www.pgdp.net (This file
was produced from images generously made available
by The Internet Archive)

*** START OF THE PROJECT GUTENBERG EBOOK THE RISING SON;

OR, THE ANTECEDENTS AND ADVANCEMENT OF THE COLORED
RACE ***
 Transcriber's Note:

Obvious typographic errors have been

corrected.
 THE

RISING SON;

OR,

THE ANTECEDENTS AND ADVANCEMENT

OF THE COLORED RACE.

BY

WM. WELLS BROWN, M. D.

AUTHOR OF “SKETCHES OF PLACES AND PEOPLE ABROAD,”

“THE
BLACK MAN,” “THE NEGRO IN THE REBELLION,”
“CLOTELLE,” ETC.

Thirteenth Thousand.
 BOSTON:
A. G. BROWN & CO., PUBLISHERS.
1882.

Entered, according to Act of Congress, in the year 1873,

By A. G. BROWN
In the Office of the Librarian of Congress, at Washington.
 PREFACE.

After availing himself of all the reliable information obtainable, the

author is compelled to acknowledge the scantiness of materials for a
history of the African race. He has throughout endeavored to give a
faithful account of the people and their customs, without concealing
their faults.
Several of the biographical sketches are necessarily brief, owing to
the difficulty in getting correct information in regard to the subjects
treated upon. Some have been omitted on account of the same
cause.
WM. WELLS BROWN.
Cambridgeport, Mass.
 Publishers’ Note to the 13th Edition.

Few works written upon the colored race have equaled in circulation
“The Rising Son.”
In the past two years the sales have more than doubled in the
Southern States, and the demand for the book is greatly on the
increase. Twelve thousand copies have already been sold; and if this
can be taken as an index to the future, we may look forward with
hope that the colored citizens are beginning to appreciate their own
authors.
WELCOME TO “THE RISING SON.”
BY ELIJAH W. SMITH.
Come forth, historian of our race,
And with the pen of Truth
Bring to our claim to Manhood’s rights,
The strength of written proof;
Draw back the curtain of the past,
And lift the ages’ pall,
That we may view the portraits grand
That hang on History’s wall!

Tell of a race whose onward tide

Was often swelled with tears;
In whose hearts bondage has not quenched
The fire of former years
When Hannibal’s resistless hosts
Wrought his imperial will,
And brave Toussaint to freedom called,
From Hayti’s vine-clad hill.

Write when, in these, our later days,

Earth’s noble ones are named,
We have a roll of honor, too,
Of which we’re not ashamed;
If, for the errors of the past,
In chains did we atone,
God, from our race’s sepulchre,
Hath rolled away the stone.

And our dear land, that long hath slept

Beneath oppression’s spell,
Welcomes the manly fortitude
That stood the test so well;
Bearing the record, blazoned o’er
With deeds of valor done,
Up to the Future’s golden door
He comes, the “Rising Son.”
The battle’s din hath passed away,
And o’er the furrowed plain
Spring, fresh and green, the tender blades
Of Freedom’s golden grain;
But eagle eyes must watch the field,
Lest the fell foe should dare
To scatter, while the sowers sleep,
Proscription’s noxious snare.

Lo! shadowy ’mid the forest-trees

Their demon forms are seen,
And lurid light of baleful eyes
Flash through the foliage green;
And till completed is the work
So gloriously begun,
A sentry true on Freedom’s walls
Stand thou, O “Rising Son!”

Go forth! the harbinger of days

More glorious than the past;
Hushed is the clash of hostile steel,
The bugle’s battle-blast;
Go, herald of the promised time,
When men of every land
Shall hasten joyfully to grasp
The Ethiope’s outstretched hand!
 CONTENTS.

PAGE
Memoir of the Author 9

CHAPTER I.
The Ethiopians and Egyptians 36

CHAPTER II.
The Carthaginians 49

CHAPTER III.
Eastern Africa 65

CHAPTER IV.
Causes of Color 78

CHAPTER V.
Causes of the Difference in Features 84

CHAPTER VI.
Civil and Religious Ceremonies 90

CHAPTER VII.
The Abyssinians 97

CHAPTER VIII.
Western and Central Africa 101
 CHAPTER IX.
The Slave-Trade 118

CHAPTER X.
The Republic of Liberia 129

CHAPTER XI.
Progress in Civilization 135

CHAPTER XII.
Hayti 140

CHAPTER XIII.
Success of Toussaint 150

CHAPTER XIV.
Capture of Toussaint 159

CHAPTER XV.
Toussaint a Prisoner in France 168

CHAPTER XVI.
Dessalines as Emperor of Hayti 173

CHAPTER XVII.
War between the Blacks and Mulattoes of Hayti 185

CHAPTER XVIII.
Christophe as King, and Pétion as President of Hayti 201

CHAPTER XIX.
Peace in Hayti, and Death of Pétion 209
 CHAPTER XX.
Boyer the Successor of Pétion in Hayti 218

CHAPTER XXI.
Insurrection, and Death of Christophe 222

CHAPTER XXII.
Union of Hayti and Santo Domingo 229

CHAPTER XXIII.
Soulouque as Emperor of Hayti 234

CHAPTER XXIV.
Geffrard as President of Hayti 236

CHAPTER XXV.
Salnave as President of Hayti 241

CHAPTER XXVI.
Jamaica 243

CHAPTER XXVII.
South America 255

CHAPTER XXVIII.
Cuba and Porto Rico 258

CHAPTER XXIX.
Santo Domingo 262

CHAPTER XXX.
Introduction of Blacks into American Colonies 265
 CHAPTER XXXI.
Slaves in the Northern Colonies 270

CHAPTER XXXII.
Colored Insurrections in the Colonies 276

CHAPTER XXXIII.
Black Men in the Revolutionary War 282

CHAPTER XXXIV.
Blacks in the War of 1812 286

CHAPTER XXXV.
The Curse of Slavery 291

CHAPTER XXXVI.
Discontent and Insurrection 296

CHAPTER XXXVII.
Growing Opposition to Slavery 319

CHAPTER XXXVIII.
Mob Law Triumphant 322

CHAPTER XXXIX.
Heroism at Sea 325

CHAPTER XL.
The Iron Age 329

CHAPTER XLI.
Religious Struggles 336
 CHAPTER XLII.
John Brown’s Raid on Harper’s Ferry 340

CHAPTER XLIII.
Loyalty and Bravery of the Blacks 342

CHAPTER XLIV.
The Proclamation of Freedom 347

CHAPTER XLV.
Blacks enlisted, and in Battle 352

CHAPTER XLVI.
Negro Hatred at the North 382

CHAPTER XLVII.
Caste and Progress 387

CHAPTER XLVIII.
The Abolitionists 393

CHAPTER XLIX.
The New Era 413

CHAPTER L.
Race Representatives.
PAGE. PAGE.
Attucks, C. 418 | Downing, G. T. 474
Aldridge, Ira. 489 | Dunn, O. J. 491
Banneker, B. 425 | Douglass, L. H. 543
Brown, I. M. 449 | Day, W. H. 499
Bell, P. A. 470 | Elliott, R. B. 403
Butler, W. F. 525 | Forten, C. L. 475
Banister, E. M. 483 | Freeman, J. J. 551
Bassett, E. D. 497 | Gaines, J. I. 450
Bell, J. M. 504 | Grimes, L. A. 534
Campbell, J. P. 446 | Garnett, H. H. 457
Clark, P. H. 520 | Greener, R. T. 542
Chester, T. M. 526 | Harper, F. E. 524
Clinton, J. J. 528 | Hayden, L. 547
Carey, M. S. 539 | Jackson, F. M. 508
Cardozo, T. W. 495 | Jones, S. T. 531
Cain, R. H. 544 | Jordan, E., Sir 481
Douglass, F. 435 | Lewis, E. 465
Delany, M. R. 460 | Langston, J. M. 447
De Mortie, L. 496 | Ransier, A. H. 510
Martin, J. S. 535 | Ruffin, G. L. 540
Nell, W. C. 485 | Still, W. 520
Purvis, C. B. 549 | Simpson, W. H. 478
Purvis, R. 468 | Smith, M’Cune 453
Pinchback, P. B. S. 517 | Smith, S. 445
Pennington, J. W. C. 461 | Smith, E. W. 552
Payne, D. A. 454 | Tanner, B. T. 530
Perry, R. L. 533 | Vashon, G. B. 476
Quinn, W. P. 432 | Wheatley, P. 423
Reason, C. L. 442 | Wayman, —— 440
Ray, C. B. 472 | Wilson, W. J. 444
Remond, C. L. 459 | Whipper, W. 493
Ruggles, D. 434 | Wears, I. C. 512
Reveles, H. R. 500 | Zuille, J. J. 473
Rainey, J. H. 507 |
 MEMOIR OF THE AUTHOR.
BY ALONZO D. MOORE.
Thirty years ago, a young colored man came to my father’s house at
Aurora, Erie County, New York, to deliver a lecture on the subject of
American Slavery, and the following morning I sat upon his knee
while he told me the story of his life and escape from the South.
Although a boy of eight years, I still remember the main features of
the narrative, and the impression it made upon my mind, and the
talk the lecture of the previous night created in our little quiet town.
That man was William Wells Brown, now so widely-known, both at
home and abroad. It is therefore with no little hesitancy that I
consent to pen this sketch of one whose name has for many years
been a household word in our land.
William Wells Brown was born in Lexington, Ky., in the year 1816.
His mother was a slave, his father a slaveholder. The boy was taken
to the State of Missouri in infancy, and spent his boyhood in St.
Louis. At the age of ten years he was hired out to a captain of a
steamboat running between St. Louis and New Orleans, where he
remained a year or two, and was then employed as office boy by
Elijah P. Lovejoy, who was at that time editor of the St. Louis Times.
Here William first began the groundwork of his education. After one
year spent in the printing office, the object of our sketch was again
let out to a captain of one of the steamboats plying on the river. In
the year 1834 William made his escape from the boat, and came
North.
He at once obtained a situation on a steamer on Lake Erie, where, in
the position of steward, he was of great service to fugitive slaves
making their way to Canada. In a single year he gave a free passage
across the lake to sixty-five fugitives. Making his home in Buffalo, Mr.
Brown organized a vigilance committee whose duties were to protect
and aid slaves, while passing through that city on their way to the
“Land of the free,” or to the eastern States. As chairman of that
committee, Mr. Brown was of great assistance to the fleeing
bondmen. The Association kept a fund on hand to employ counsel in
case of capture of a fugitive, besides furnishing all with clothing,
shoes, and whatever was needed by those who were in want.
Escaping from the South without education, the subject of our
sketch spent the winter nights in an evening school and availed
himself of private instructions to gain what had been denied him in
his younger days.
In the autumn of 1843, he accepted an agency to lecture for the
Anti-slavery Society, and continued his labors in connection with that
movement until 1849; when he accepted an invitation to visit
England. As soon as it was understood that the fugitive slave was
going abroad, the American Peace Society elected him as a delegate
to represent them at the Peace Congress at Paris.
Without any solicitation, the Executive Committee of the American
Anti-slavery Society strongly recommended Mr. Brown to the friends
of freedom in Great Britain. The President of the above Society gave
him private letters to some of the leading men and women in
Europe. In addition to these, the colored citizens of Boston held a
meeting the evening previous to his departure, and gave Mr. Brown
a public farewell, and passed resolutions commending him to the
confidence and hospitality of all lovers of liberty in the mother-land.
Such was the auspices under which this self-educated man sailed for
England on the 18th of July, 1849.
Mr. Brown arrived in Liverpool, and proceeded at once to Dublin,
where warm friends of the cause of freedom greeted him. The land
of Burke, Sheridan, and O’Connell would not permit the American to
leave without giving him a public welcome. A large and enthusiastic
meeting held in the Rotunda, and presided over by James Haughton,
Esq., gave Mr. Brown the first reception which he had in the Old
World.
After a sojourn of twenty days in the Emerald Isle, the fugitive
started for the Peace Congress which was to assemble at Paris. The
Peace Congress, and especially the French who were in attendance
at the great meeting, most of whom had never seen a colored
person, were somewhat taken by surprise on the last day, when Mr.
Brown made a speech. “His reception,” said La Presse, “was most
flattering. He admirably sustained his reputation as a public speaker.
His address produced a profound sensation. At its conclusion, the
speaker was warmly greeted by Victor Hugo, President of the
Congress, Richard Cobden, Esq., and other distinguished men on the
platform. At the soirée given by M. de Tocqueville, the Minister for
Foreign Affairs, the American slave was received with marked
attention.”
Having spent a fortnight in Paris and vicinity, viewing the sights, he
returned to London. George Thompson, Esq., was among the first to
meet the fugitive on his arrival at the English metropolis. A few days
after, a very large meeting, held in the spacious Music Hall, Bedford
Square, and presided over by Sir Francis Knowles, Bart., welcomed
Mr. Brown to England. Many of Britain’s distinguished public speakers
spoke on the occasion. George Thompson made one of his most
brilliant efforts. This flattering reception gained for the fugitive
pressing invitations from nearly all parts of the United Kingdom.
He narrates in his “Three Years in Europe,” many humorous incidents
that occurred in his travels, and of which is the following:
“On a cold winter’s evening, I found myself seated before the fire,
and alone, in the principal hotel in the ancient and beautiful town of
Ludlow, and within a few minutes’ walk of the famous old castle
from which the place derives its name. A long ride by coach had so
completely chilled me, that I remained by the fire to a later hour
than I otherwise would have.
“‘Did you ring, sir?’ asked the waiter, as the clock struck twelve.
“‘No,’ I replied; ‘but you may give me a light, and I will retire.’
“I was shown to my chamber, and was soon in bed. From the weight
of the covering, I felt sure that the extra blanket which I had
requested to be put on was there; yet I was shivering with cold. As
the sheets began to get warm, I discovered, to my astonishment,
that they were damp—indeed, wet. My first thought was to ring the
bell for the servant, and have them changed; but, after a moment’s
consideration, I resolved to adopt a different course. I got out of
bed, pulled the sheets off, rolled them up, raised the window, and
threw them into the street. After disposing of the wet sheets, I
returned to bed, and got in between the blankets, and lay there
trembling with cold till Morpheus came to my relief.
“The next morning I said nothing about the sheets, feeling sure that
the discovery of their loss would be made by the chambermaid in
due time. Breakfast over, I visited the ruins of the old castle, and
then returned to the hotel, to await the coach for Hereford. As the
hour drew near for me to leave, I called the waiter, and ordered my
bill. ‘Yes, sir, in a moment,’ he replied, and left in haste. Ten or
fifteen minutes passed away, and the servant once more came in,
walked to the window, pulled up the blinds, and then went out.
“I saw that something was afloat; and it occurred to me that they
had discovered the loss of the sheets, at which I was pleased; for
the London newspapers were, at that time, discussing the merits
and the demerits of the hotel accommodations of the kingdom, and
no letters found a more ready reception in their columns than one
on that subject. I had, therefore, made up my mind to have the wet
sheets put in the bill, pay for them, and send the bill to the Times.
“The waiter soon returned again, and, in rather an agitated manner,
said, ‘I beg your pardon, sir, but the landlady is in the hall, and
would like to speak to you.’ Out I went, and found the finest
specimen of an English landlady that I had seen for many a day.
There she stood, nearly as thick as she was tall, with a red face
garnished around with curls, that seemed to say, ‘I have just been
oiled and brushed.’ A neat apron covered a black alpaca dress that
swept the floor with modesty, and a bunch of keys hung at her side.
O, that smile! such a smile as none but an adept could put on.
However, I had studied human nature too successfully not to know
that thunder and lightning were concealed under that smile, and I
nerved myself for the occasion.
“‘I am sorry to have to name it, sir,’ said she; ‘but the sheets are
missing off your bed.’
“‘O, yes,’ I replied; ‘I took them off last night.’
“‘Indeed!’ exclaimed she; ‘and what did you do with them?’
“‘I threw them out of the window,’ said I.
“‘What! into the street?’
“‘Yes; into the street,’ I said.
“‘What did you do that for?’
“‘They were wet; and I was afraid that if I left them in the room they
would be put on at night, and give somebody else a cold.’
“‘Then, sir,’ said she, ‘you’ll have to pay for them.’
“‘Make out your bill, madam,’ I replied, ‘and put the price of the wet
sheets in it, and I will send it to the Times, and let the public know
how much you charge for wet sheets.’
“I turned upon my heel, and went back to the sitting-room. A
moment more, and my bill was brought in; but nothing said about
the sheets, and no charge made for them. The coach came to the
door; and as I passed through the hall leaving the house, the
landlady met me, but with a different smile.
“‘I hope, sir,’ said she, ‘that you will never mention the little incident
about the sheets. I am very sorry for it. It would ruin my house if it
were known.’ Thinking that she was punished enough in the loss of
her property, I promised not to mention the name of the house, if I
ever did the incident.
“The following week I returned to the hotel, when I learned the fact
from the waiter that they had suspected that I had stolen the
sheets, and that a police officer was concealed behind the hall door,
on the day that I was talking with the landlady. When I retired to
bed that night, I found two jugs of hot water in the bed, and the
sheets thoroughly dried and aired.
“I visited the same hotel several times afterwards, and was
invariably treated with the greatest deference, which no doubt was
the result of my night with the wet sheets.”
In 1852, Mr. Brown gave to the public his “Three Years in Europe,” a
work which at once placed him high as an author, as will be seen by
the following extracts from some of the English journals. The Eclectic
Review, edited by the venerable Dr. Price, one of the best critics in
the realm, said,—“Mr. Brown has produced a literary work not
unworthy of a highly-cultivated gentleman.”
Rev. Dr. Campbell, in the British Banner, remarked: “We have read
Mr. Brown’s book with an unusual measure of interest. Seldom,
indeed, have we met with anything more captivating. A work more
worthy of perusal has not, for a considerable time, come into our
hands.”
“Mr. Brown writes with ease and ability,” said the Times, “and his
intelligent observations upon the great question to which he has
devoted and is devoting his life will command influence and respect.”
The Literary Gazette, an excellent authority, says of it, “The
appearance of this book is too remarkable a literary event, to pass
without a notice. At the moment when attention in this country is
directed to the state of the colored people in America, the book
appears with additional advantage; if nothing else were attained by
its publication, it is well to have another proof of the capability of the
negro intellect. Altogether, Mr. Brown has written a pleasing and
amusing volume, and we are glad to bear this testimony to the
literary merit of a work by a negro author.”
The Glasgow Citizen, in its review, remarked,—“W. Wells Brown is no
ordinary man, or he could not have so remarkably surmounted the
many difficulties and impediments of his training as a slave. By dint
of resolution, self-culture, and force of character, he has rendered
himself a popular lecturer to a British audience, and a vigorous
expositor of the evils and atrocities of that system whose chains he
has shaken off so triumphantly and forever. We may safely
pronounce William Wells Brown a remarkable man, and a full
refutation of the doctrine of the inferiority of the negro.”
The Glasgow Examiner said,—“This is a thrilling book, independent
of adventitious circumstances, which will enhance its popularity. The
author of it is not a man, in America, but a chattel,—a thing to be
bought, and sold, and whipped; but in Europe, he is an author, and
a successful one, too. He gives in this book an interesting and
graphic description of a three years’ residence in Europe. The book
will no doubt obtain, as it well deserves, a rapid and wide
popularity.”
In the spring of 1853, the fugitive brought out his work, “Clotelle; or,
the President’s Daughter,” a book of nearly three hundred pages,
being a narrative of slave life in the Southern States. This work
called forth new criticisms on the “Negro Author” and his literary
efforts. The London Daily News pronounced it a book that would
make a deep impression; while The Leader, edited by the son of
Leigh Hunt, thought many parts of it “equal to anything which had
appeared on the slavery question.”
The above are only a few of the many encomiums bestowed upon
our author. Besides writing his books, Mr. Brown was also a regular
contributor to the columns of The London Daily News, The Liberator,
Frederick Douglass’ Paper, and The National Anti-slavery Standard.
When we add, that in addition to his literary labors, Mr. Brown was
busily engaged in the study of the medical profession, it will be
admitted that he is one of the most industrious of men. After
remaining abroad nearly six years, and travelling extensively through
Great Britain and on the continent, he returned to the United States
in 1854, landing at Philadelphia, where he was welcomed in a large
public meeting presided over by Robert Purvis, Esq.
On reaching Boston, a welcome meeting was held in Tremont
Temple, with Francis Jackson, Esq., in the chair, and at which
Wendell Phillips said,—“I rejoice that our friend Brown went abroad;
I rejoice still more that he has returned. The years any thoughtful
man spends abroad must enlarge his mind and store it richly. But
such a visit is to a colored man more than merely intellectual
education. He lives for the first time free from the blighting chill of
prejudice. He sees no society, no institution, no place of resort or
means of comfort from which his color debars him.
“We have to thank our friend for the fidelity with which he has, amid
many temptations, stood by those whose good name religious
prejudice is trying to undermine in Great Britain. That land is not all
Paradise to the colored man. Too many of them allow themselves to
be made tools of the most subtle of their race. We recognize, to-
night, the clear-sightedness and fidelity of Mr. Brown’s course
abroad, not only to thank him, but to assure our friends there that
this is what the Abolitionists of Boston endorse.”
Mr. Phillips proceeded:—“I still more rejoice that Mr. Brown has
returned. Returned to what? Not to what he can call his ‘country.’
The white man comes ‘home.’ When Milton heard, in Italy, the sound
of arms from England, he hastened back—young, enthusiastic, and
bathed in beautiful art as he was in Florence. ‘I would not be away,’
he said, ‘when a blow was struck for liberty.’ He came to a country
where his manhood was recognized, to fight on equal footing.