In the late 18th century, the formerly enslaved Black man Benjamin Honey marries the white Irish woman Patience, and the two of them settle on an island off the coast of Maine. Life on the island is difficult, with little food, but through persistence, Benjamin manages to grow an orchard of apple trees on the island, earning it the name Apple Island.
The settlement on Apple Island continues to grow, with other families joining it. Then one day, a little over 20 years from the initial settlement, a hurricane strikes and causes a massive flood across the island. All of the members of the Honey family take shelter by climbing a tree, but the flood soon sweeps the tree away with the family members in it. As the water rises, Patience’s head goes under, but she keeps her arm in the air, holding a patched-together flag that represents the island. The water never submerges the flag. By the time the flood waters recede, many Apple Islanders have died, the buildings have been wiped away, and the original orchard is gone. But the Honeys don’t leave the island, and soon, they rebuild.
Many years later, in the early 20th century, Esther Honey is the great-granddaughter of Benjamin and Patience. She lives in a shack with her son, Eha, and his children, Ethan, Tabitha, and Charlotte. Eha built the shack himself, under the guidance of Zachary Hand, an old man who lives on the far side of the island and spends most of his time carving depictions of Bible stories onto a hollow tree. Eha’s father is Esther’s own father—he raped her, and Esther considered drowning Eha on the day of his birth. Instead, she decided to keep Eha but kill her father, pushing him off a bluff when he wasn’t paying attention.
Other residents of the island include Candace and Theophilus. They are rumored to be siblings in an incestuous relationship and who have several children, including a daughter named Rabbit, who eats unusual things. There’s also Iris and Violet, who are unmarried sisters who live together with several adopted children.
Things begin to change on Apple Island one day when the white mainlander Matthew Diamond sets up a school for the islanders. Matthew Diamond holds some racist views but also seems to have a genuine interest in the islanders, encouraging Ethan in particular to study art. Ethan begins making drawings in his free time, and eventually Matthew Diamond arranges for Ethan to stay with his friend Thomas Hale in Massachusetts, where Ethan will be able to draw the landscapes and take art classes.
Ethan agrees to go to stay with Thomas Hale, but he finds that when he gets there, Thomas Hale doesn’t let him inside the house, forcing him to stay in the barn. Shortly after arriving, Ethan begins to bond with Thomas Hale’s Irish maid, Bridget. She is fascinated by Ethan’s drawing, and he takes an interest in her after she starts singing an Irish lullaby that he recognizes from Apple Island (since the Honeys all have Irish heritage through Patience). Shortly after Ethan does a portrait of Bridget, the two of them start having sex. One day, Bridget sees drawings and photos of Ethan’s relatives and realizes, based on the dark skin of his relatives, that Ethan isn’t fully white but is actually mixed race. Shortly afterwards, Thomas Hale witnesses Bridget going to see Ethan in his barn at night and throws Ethan out of his home.
Ethan disappears, never returning home but possibly becoming an artist who moves to France. Meanwhile, a pregnant Bridget comes to Apple Island to see Esther, who invites Bridget into the Honey family. Committees from the government, including even the governor himself, have been coming to see Apple Island and evaluate the residents. Although Matthew Diamond believes the plan is to renovate the island, in fact, the government ends up evicting all of the original Apple Islanders. This is mostly due to popular eugenicist ideas, which suggest the Apple Islanders are “inferior.”
The eviction notices are traumatic for many of the Apple Islanders. A sheriff and deputy come to take away Rabbit and put her in a school for the “feebleminded.” Candace tries to stop them, but in the ensuing struggle, the sheriff and deputy accidentally kill Rabbit, causing grief across the island.
Eventually, most islanders come to accept their fate. Eha takes apart the Honey shack piece by piece, preserving the pieces to take on a raft to the mainland. The Honeys even let the arthritic Esther stay in her rocking chair as they tie her in place in the raft. Iris and Violet leave on a separate raft with their adopted children. They all go to the mainland to start new lives.
When the white mainlanders come to destroy all evidence of the original Apple Island settlers, digging up graves and burning houses, the only remaining islander is Zachary Hand. He walks out into the water while holding up a flag.